AI Voice Agents for Business: Architecture, Cost and What Actually Works

An ai voice agent is a phone system that listens, understands and speaks in real time, replacing the menu tree with an actual conversation. In 2026 a production build takes one of two shapes: a cascaded pipeline (streaming speech recognition, then a language model, then speech synthesis) or a single speech-to-speech model that does all three. The binding constraint is latency. Human conversation runs on gaps of roughly 100 to 200 milliseconds, so a second of dead air reads as a machine. All-in cost lands between about $0.10 and $0.35 per minute once telephony, models and platform fees are counted.

The rule that changed on the day this published

From 2 August 2026, Article 50 of the EU AI Act applies. It requires AI systems built to interact directly with people to be designed so the person knows they are dealing with an AI, unless that is already obvious to a reasonably well-informed observer. Nothing about a good synthetic voice on a phone line is obvious any more, so the disclosure has to be in the call. The European Commission's guidance says it must be clear, distinguishable, delivered at the start, and accessible. Article 99(4)(g) sets the penalty at up to 15 million euro or 3% of worldwide annual turnover, whichever is higher.

We have been retrofitting that line into deployed agents for six weeks. It is two seconds of audio and it costs conversion: callers hang up more often when you tell them. They also complain less, and in one client's escalation log the "I did not know it was a robot" category went to zero. Build it in on day one and measure against it.

The rest of this is the engineering. The category-level view of where conversational systems fit in a support organisation is already covered in our 2026 guide to enterprise conversational AI and the definitive enterprise chatbot guide. This post is the phone line specifically: the milliseconds, the stack, the bill, and the calls you should refuse to automate.

IVR vs AI phone agent: what actually changed under the hood

The old interactive voice response system was a decision tree with audio attached, matching keypresses or a tiny grammar against a fixed script. It could not handle "I think my invoice is wrong but actually I also moved house" because that sentence is not a node.

The ivr vs ai phone agent question is not about intelligence. It is about where state lives. In an IVR, state is the node you are standing on. In an ai voice agent, state is a conversation history plus tool calls the model can make against your systems, and it re-plans on every turn. That is why one handles interruption and the other does not.

The last row is the honest one. An IVR is instant because it is playing a file. Every ivr vs ai phone agent comparison that skips latency is selling you something. The whole engineering problem is buying back the time you just spent on understanding.

The latency budget is the product

Levinson and Torreira's 2015 analysis in Frontiers in Psychology put the modal gap between conversational turns at 100 to 200 milliseconds, while the language production that fills that gap takes upwards of 600 milliseconds on its own. People start planning their reply before you finish speaking. Stivers and colleagues found the same pattern across ten languages in their 2009 PNAS study, with cross-language averages varying by no more than about 250 milliseconds. That is the bar your voice ai is judged against, unconsciously, by every caller.

You will not hit 200 ms over a phone network. What you can do is stay under the threshold where a pause stops reading as thinking and starts reading as broken. We hold our builds to 800 ms p50 and 1.2 s p95, measured from end of caller speech to first audio out.

Here is the budget we work to on a cascaded stack. Cited figures are vendor-published; the network and model rows are our own measurements from US and EU production deployments.

Two things fall out of that table. End-of-turn detection is the largest controllable line, which makes it worth more attention than model choice: a naive 800 ms silence timeout, a common default, adds nearly a full second before the pipeline even starts. Prompt length is also a latency decision, because every thousand tokens of instructions pushes time-to-first-token out and callers feel it. The cheapest latency win we have found is not a faster model. It is halving the system prompt and moving the removed material into tools the model calls only when it needs them.

Turn-taking, barge-in and being interrupted mid-sentence

A properly built agent can be interrupted, and if yours cannot, treat that as a defect. Barge-in means the agent keeps listening while it speaks, detects that the caller has started talking, cancels the audio it is playing, drops the unspoken remainder of its turn from context, and re-plans.

The hard part is not detecting sound. It is deciding whether the sound is an interruption. "Mm-hmm" and "right" are backchannels; a caller saying them is telling you to keep going. Cutting yourself off on every backchannel makes the agent feel jumpy, which is worse than being slightly slow.

The open-source frameworks have converged on the same answer. LiveKit Agents ships a turn detector that encodes caller audio directly and combines semantic signal with acoustic cues like intonation and rhythm, running locally so it adds no network hop. Pipecat ships its Smart Turn analyser for the same reason. Deepgram folded turn detection into the recognition model itself with Flux, which reached multilingual general availability across ten languages on 29 April 2026.

On real phone calls we still tune three parameters per deployment: the minimum speech duration that counts as a barge-in, the backchannel word list, and whether the agent resumes or abandons its interrupted sentence. Those settings are different for an ai call center voice agent handling billing disputes than for an outbound qualification call, because angry callers interrupt differently from curious ones. This is also the honest answer to what voice ai works best for outbound sales calls: the one whose barge-in handling survives a prospect talking over the opening line.

The speech recognition problem nobody puts in the demo

Every vendor demo is recorded on a good microphone in a quiet room by someone with an accent the model was trained on. Your callers are on a phone, in a car, in a warehouse, with a cold.

Two structural problems make this harder than the demo suggests. Telephony audio is narrowband, historically 300 Hz to 3.4 kHz, which strips out the high-frequency energy that distinguishes fricatives like "s" and "f". And recognition accuracy is not evenly distributed across speakers. Koenecke and colleagues, publishing in PNAS in 2020, measured five commercial recognition systems and found an average word error rate of 0.35 for Black speakers against 0.19 for white speakers on the same task, with the gap widest for speakers using more African American Vernacular English features. Those specific systems and that specific vintage have moved on considerably since 2020, but the mechanism has not: models are best on the accents best represented in their training data, and phone audio amplifies the difference.

Four things we do about it. Keyword boosting on the terms that matter (product names, drug names, street names in the service area) so one misrecognised word does not cascade. Read-back confirmation on anything consequential instead of trusting a single transcript. An explicit failure path: two consecutive low-confidence turns and the caller goes to a human, no third attempt. And testing on recordings from the client's own call archive rather than the team's voices, because internal testing over-represents the accents already in the building.

If a meaningful share of your callers speak a language the stack handles badly, route those calls to people. That is not a limitation to ship around.

Cascaded pipeline or speech-to-speech

The cascaded design (recognition, then model, then synthesis) is what most production systems still run and what we default to. Any component can be swapped, you get a text transcript free at every turn, and deterministic guardrails can sit between the stages.

Speech-to-speech models skip the text hop. OpenAI released GPT-Live-1 and GPT-Live-1 mini on 8 July 2026, full-duplex models that listen and speak at the same time and handle turn-taking natively, with the mini replacing Advanced Voice Mode as the ChatGPT default. Prosody and interruption handling are both better than a cascaded stack, and the latency floor is lower because there are fewer hops.

The tradeoffs are why we have not moved most clients across. You lose the intermediate transcript as a control point, so guardrails live inside the prompt instead of between stages. You are locked to one vendor's voice, reasoning and outage. Cost is harder to forecast on audio tokens than on minutes. And compliance review is harder with no text artefact to audit.

Our rule as of August 2026: cascaded for anything regulated, anything with a hard cost ceiling, or anything that has to log a defensible transcript. Speech-to-speech for consumer-facing, low-stakes, high-volume calls where naturalness pays for itself. That line will move.

Telephony: do you need a new number?

No. A voice agent does not require a new phone number, and in most builds it should not have one, because a new number breaks caller ID recognition and any print or vehicle signage you already paid for.

There are three ways to attach an agent to a line you already own. Conditional call forwarding is the cheapest: your number forwards to the agent after a set ring count or when the line is busy, so humans answer first and the agent catches overflow. That is how most packaged products do it, including the jobber ai receptionist, which documents working with your existing number via call forwarding. Second, port the number to a programmable carrier such as Twilio or Telnyx and route it in software. Third, and best if you already run a PBX or contact centre platform, terminate a SIP trunk into the agent and leave your numbering plan untouched.

We start clients on conditional forwarding for the first four weeks. It reverses in about a minute, which matters more than elegance when the phone line is how the business gets paid. Port the number once the agent has earned it.

What an AI voice agent actually costs per minute

The advertised price is never the price. Platform vendors quote their own margin line and pass everything else through, which is why "$0.05 a minute" and a $0.28 bill are both true statements about the same product.

Here is the stack priced from vendor pages as of 2 August 2026.

Retell's own worked example on default settings comes to roughly $0.11 per minute. Swap in a frontier model and premium voices and you are at $0.25 to $0.30 before add-ons. That $0.10 to $0.30 band is what we quote clients, and it is where nearly every deployment we have priced lands.

Speech-to-speech bills differently. OpenAI's realtime audio pricing is $32 per million input tokens and $64 per million output tokens for gpt-realtime, and $10 and $20 for the mini, with cached input at $0.40 per million. Assistant audio consumes tokens faster than caller audio, so long calls skew expensive unless prompt caching is working. Instrument token spend per call from day one rather than at the first invoice.

Two costs the table hides. Building and evaluating the agent is engineering time, typically six to ten weeks for a first production line. And the deflection maths only works if the agent resolves calls rather than delaying them: a call it handles for ninety seconds and then transfers costs more than the transfer alone would have. We break that down in the batch post on what customer service automation deflects and what it costs.

Handing off without making the caller repeat themselves

The most damaging failure in voice deployments is the cold transfer. Caller explains the problem for two minutes, gets passed to a human, and the human says "how can I help?"

A warm handoff is engineering, not prompting. The agent places the transfer over SIP REFER or an equivalent bridged transfer so the call leg survives. It pushes a structured context payload (caller identity, verified fields, intent classification, transcript, tool calls) into the agent desktop before the human picks up, which means a real integration into your CRM or ticketing system rather than a screen pop with a phone number on it. And the human's opening line is scripted to confirm rather than ask: "I have you down as calling about the December invoice, is that right?"

Build the fallback too. If the transfer fails or the queue is over threshold, the agent takes a callback instead of dumping the caller into hold music. Callers forgive "everyone is busy, I will have Maria call you back inside the hour" far more readily than five minutes of ringing.

Stopping it from inventing an answer

Voice makes hallucination worse than text, because a caller cannot scroll back and there is no visible source to check. A confident wrong sentence spoken aloud is indistinguishable from a right one.

Prompting alone does not fix this. Removing the model's opportunity to guess does:

• Answer from tools, not memory. Prices, appointment slots, order status and policy come from a function call against a system of record. If the call fails, the agent says the system is unavailable and offers a callback.
• Constrain retrieval to a closed corpus of approved answers. If the question falls outside it, the correct behaviour is a transfer.
• Make refusal cheap and specific. "I do not have that in front of me, let me get someone who does" is a good outcome, and it has to be trained and evaluated for, because models lean toward being helpful.
• Never let the model state a number it did not receive from a tool. Any evaluation call where a figure appears in the audio but not in a tool response is a hard fail.

The measurable version is a grounded-answer rate. We hold production agents above 98% of factual claims traceable to a tool response, sampled weekly. It is a slower agent than one allowed to improvise, and it is the only kind we will put on a client's main line.

Compliance: outbound calls, card details and health data

Placing outbound calls with a synthetic voice is legal in the United States, and heavily conditioned. The FCC's Declaratory Ruling of 8 February 2024 (FCC 24-17, CG Docket 23-362) held that AI-generated voices are "artificial" under the Telephone Consumer Protection Act, which pulls AI calls into the existing consent, identification and opt-out regime for artificial and prerecorded voice calls. The FCC then proposed a dedicated AI-generated call definition with an in-call disclosure requirement in its July 2024 rulemaking, published in the Federal Register on 10 September 2024. That proposal has not been finalised, so the operative rule today is still the 2024 declaratory ruling plus the underlying TCPA.

States moved anyway. California's AB 2905, effective 1 January 2025, requires prerecorded messages using an artificial voice to disclose that fact at the start of the call. Utah's AI Policy Act, in force since 1 May 2024 and narrowed by SB 226 in 2025, requires disclosure when a person interacts with generative AI in regulated occupations and high-risk interactions. The FCC also adopted a further rulemaking on 20 May 2026 proposing to expand know-your-upstream-provider and STIR/SHAKEN obligations, which lands on the carriers your outbound traffic passes through rather than on you, but changes what your carrier asks of you.

In the EU, Article 50 of the AI Act now applies as above. The UK has no AI Act equivalent; automated calling sits under PECR and UK GDPR, enforced by the ICO, with Ofcom handling silent and abandoned call rules.

Card payments are the clearest case of "do not automate this by default." PCI DSS v4.0.1, published on 11 June 2024, is the current version of the standard, with its new requirements effective from 31 March 2025. Pausing the recording while a caller reads their card number does not remove your telephony, workstations or network from the cardholder data environment. The architectural answer is DTMF masking: the caller keys the digits, the tones are intercepted and flattened before they reach your agent or your recording, and the digits go straight to the payment provider. Your voice agent should never hold a primary account number in context. Ours hand off to a payment flow and get back a token and a result code.

Health data is the same problem with a different acronym, and on a patient call the harder question is not the data path but whether the agent is allowed to answer at all. We set that boundary out in conversational AI in healthcare. An ai receptionist hipaa question is really a question about every hop: the recognition vendor, the model vendor, the synthesis vendor, the telephony carrier and the recording store each touch protected health information and each needs a business associate agreement. Most major cloud and model vendors will sign one for enterprise or zero-retention endpoints. Consumer tiers will not. Getting an ai receptionist hipaa posture right is procurement before it is engineering, which is why healthcare voice builds take longer than the demo suggests. If compliance pushes you toward keeping the model inside your own perimeter, our guide to private LLM hosting covers when that is worth the operational load.

Which call types should never reach a voice agent

We build this list with the client before we build the agent, and it goes in the routing layer, not the prompt, because a prompt can be talked out of a rule.

The last two are the ones vendors leave out. Both cost you a little containment rate and buy back a lot of goodwill.

Where this works, and what buyers call it

The category has half a dozen names depending on who is buying, and the names matter because they change what gets built. Practice managers ask for an ai medical receptionist. Clinics ask for an ai receptionist for medical office scheduling. Trades ask for ai receptionist software for plumbers and mostly find packaged products. Everyone else types ai virtual receptionist, ai voice receptionist or ai calling receptionist software into a search box and lands in a slightly different market. The pipeline underneath is identical. The integration is what differs, and the integration is where the project time goes.

What each search term actually resolves to, and whether we would buy or build it:

• ai receptionist software: calendar write access and a spam filter. Buy.
• ai receptionist software for plumbers: job type, address capture, dispatch queue. Buy.
• ai medical receptionist: practice management system read and write. Buy if your PMS has a partner integration.
• ai receptionist for medical office cover: the same, plus signed agreements down the whole stack. Build if your PMS is unusual.
• ai dental receptionist: chair time, hygienist versus dentist, recall cycles. Build, or configure heavily.
• ai receptionist for law firms: matter intake with a conflicts check before anything else. Build.
• ai voice receptionist for out-of-hours cover: forwarding rules and a callback queue. Buy.
• best ai virtual receptionist for lead qualification: CRM write, scoring rules, routing to a rep. Buy, then replace when the routing gets specific.
• ai call center voice agent: contact centre platform integration, warm transfer, reporting. Build.
• voice ai recruiter: ATS integration and an audit trail per candidate. Build.

Three of those deserve the detail.

Front desk, trades and out-of-hours

An ai receptionist that answers in two rings at 9pm beats a voicemail box by a distance, and the calls are short, structured and mostly about booking. Home services is now a packaged category: the jobber ai receptionist answers calls and texts around the clock, books jobs and captures leads for plumbers, HVAC firms, landscapers and cleaning companies, and is included on its Plus plan. If your workflow already lives in a suite like that, buy the bundled ai receptionist software before you build anything. An ai voice receptionist only justifies a custom build when the integration or the compliance requirement is yours alone. Most of the ai calling receptionist software on the market handles the generic case perfectly well, and we tell prospects so. We have talked two trades clients out of a custom project in the last year on exactly that basis: off-the-shelf ai receptionist software for plumbers already did dispatch capture better than a six-week build would have, and neither of them needed the ai calling receptionist software to do anything their field service suite could not.

Clinics, dental and regulated intake

Dental is a real sub-market because the scheduling logic is unusual: chair time, hygienist versus dentist, recall cycles, insurance verification. An ai dental receptionist that cannot see chair availability is a message-taker with a nicer voice, and every ai dental receptionist demo we have been shown skips that integration. The same is true of an ai medical receptionist that cannot read the practice management system, and it is why an ai receptionist for medical office cover so often stalls at the integration rather than the conversation. Layer the ai receptionist hipaa requirements on top and the vendor shortlist shrinks fast. When buyers ask which is the best ai receptionist, the honest answer is that the ranking is decided by which one integrates with the system you already run, not by voice quality in the demo. The best ai receptionist for a single-site clinic is usually the one bundled with its own software.

For an ai receptionist for law firms, intake is the value and conflicts checking is the constraint. The agent takes the matter type, the opposing party and the jurisdiction, and it must not say anything resembling advice. We build an ai receptionist for law firms with a hard topic classifier sitting in front of the model, not a prompt instruction, because a prompt can be argued with.

Contact centres, outbound and screening

An ai call center voice agent is a different animal from a front desk. It has to integrate with the contact centre platform, respect existing queues and skills routing, and report into the same dashboards the workforce team already uses. Containment rate is the metric, and an ai call center voice agent that reports a high containment rate while quietly abandoning callers is worse than no agent.

Outbound is its own discipline. The question of what voice ai works best for outbound sales calls has a dull answer: the one with the best interruption handling and the shortest opening line, because outbound calls are lost in the first four seconds. Anyone benchmarking what voice ai works best for outbound sales calls on transcript quality is measuring the wrong thing. For qualification, buyers looking for the best ai virtual receptionist for lead qualification should test on their own worst recordings, since products separate on a bad line rather than a clean one. There is no single best ai receptionist for that job: the best ai virtual receptionist for lead qualification is whichever one writes cleanly into your CRM and routes to a rep who can take the call now. A voice ai recruiter running first-round screening is the same pattern with a different script, and a voice ai recruiter carries employment-law exposure a sales call does not, which we cover in the batch post on AI hiring law changes. Email outbound separates on the same axis. The questions that expose a weak cold email agency are infrastructure questions, not copy ones.

How we scope a build

A first production voice line takes six to ten weeks with us, and the shape is consistent. Two weeks on call taxonomy and integrations, which means listening to fifty real recordings and deciding what the agent will and will not attempt. Three to four weeks building the pipeline, tools and handoff. Two weeks of evaluation against recorded calls with a scored rubric. Then a staged rollout: overflow only, then after-hours, then daytime primary, with a rollback that takes one config change.

What kills timelines is never the model. It is the integration into the system of record, and the discovery in week three that appointment availability is not queryable in real time.

For the wider view of scope, pricing and vendor questions before you commit, our guide to hiring an AI agent development partner sets out what to ask, and the AI chatbot development services guide covers the text-channel equivalent. XOVO builds the phone layer on the same conversational core as our AI chatbot product, so knowledge base, tools and escalation rules are shared across voice and chat rather than maintained twice. The chatbot building and integration service is where the integration work lives.

If your phone line is the first thing a customer touches, treat it as production infrastructure. Measure p95 turn latency, grounded-answer rate, transfer rate and the reason codes behind every escalation, weekly. An ai voice agent that nobody measures degrades quietly, and the phone is the last place to find that out late.

Book a free AI audit and we will listen to a sample of your real calls and tell you which of them an agent should never take. To start with the text channel instead, the AI chatbot product page has the deployment options.