AI voice agents: what they do well, and where they still fail

The demos are good enough now that the interesting question is no longer “can it hold a conversation”. It can. The interesting questions are which conversations, what it costs when it goes wrong, and what has to be engineered around it so the failures are graceful.

What is actually happening on the call

Four things run in a loop, and the loop has a hard time budget.

That budget is why architecture matters more than model choice. Every component has to stream, and each hop between services spends milliseconds you cannot get back. An agent that thinks brilliantly but replies in 1.5 seconds loses to one that thinks adequately in 400 milliseconds.

Where they genuinely work

The pattern is consistent: bounded conversations with a clear success condition, repeated at volume.

Where they still fail

• Emotionally charged calls. A complaint, a bereavement, a cancellation someone is angry about. The agent can be word-perfect and it will still be the wrong choice.
• Genuinely open-ended discovery. Complex sales discovery, technical diagnosis without a decision tree, anything where the right next question depends on judgement rather than a rule.
• Heavy accents on a poor line. Recognition accuracy degrades with compression and background noise, and it degrades unevenly across accents, which is a fairness problem as well as a quality one.
• Anything with a hard consequence and no human check. Never let an agent take an irreversible action (a cancellation, a refund, a dispatch) without a confirmation path.
• Long, information-dense answers. Callers cannot hold six spoken facts in memory. If the answer needs a table, send a text and say so.

Design decisions that separate good from bad

Disclose that it is an AI

Say it in the first sentence. Several jurisdictions now require it, and it costs less than people expect; callers are frustrated by being stuck, not by talking to software that actually helps. Concealing it and being found out is far more expensive than saying it up front.

Make the escape hatch obvious

“Say ‘agent’ at any time and I'll put you through.” Then honour it immediately, first time. The most damaging pattern in the whole category is an agent that deflects a request for a human; it converts mild frustration into a complaint.

Transfer with context

When the handoff happens, the human should receive the transcript and everything already collected. Making the caller repeat what they just said tells them the previous three minutes were wasted, and it is the moment most people give up on the brand rather than the bot.

Constrain what it can say about price and promises

Prices, availability, and commitments should come from a data source the agent reads, not from a model's memory of your prompt. An agent that improvises a discount has made an offer your business may have to honour.

Instrument every call

Transcript, structured summary, outcome, and whether it transferred. Without that you cannot tell a working agent from a broken one, and the failure mode is silent: it sounds fine and books nothing.

Outbound is harder than inbound

Inbound has an enormous advantage: the person chose to call. Outbound AI has to earn the first ten seconds against a stranger who did not.

Everything from the deliverability side applies with more force, not less. The number still needs to be owned, signed at Attestation A, and carrying a registered name, or the call is labelled before the agent says a word. Consent rules apply exactly as they do to human calls: an AI agent is automated technology under the TCPA, which raises the consent bar rather than sidestepping it.

Where outbound AI works well is the narrow band of calls the recipient is expecting: appointment reminders, confirmations, follow-ups on something they initiated. Cold outbound AI is technically possible, legally fraught, and it burns numbers fast.

What it costs and how to think about it

AI voice is billed per minute of conversation, on top of the underlying call. The right comparison is not against a human's hourly rate; it is against the calls you currently do not answer at all.

For most small businesses, the honest business case is missed calls. A clinic or a restaurant that misses a meaningful share of calls during its busiest two hours is losing bookings to whoever answers next. An agent that captures even part of that pays for itself immediately, and it costs nothing on a quiet day when it handles no calls, which is the real advantage of per-minute billing over a night-shift salary.

For larger operations the case is concurrency and consistency, not cost: ten simultaneous callers answered on the first ring, asked the same qualifying questions in the same order, every time.

A sensible way to start

1. Pick one bounded job. After-hours answering is the usual best first choice, because the baseline is voicemail and almost anything beats it.
2. Write down what the agent must never say or promise before you write what it should say.
3. Connect one real data source, usually the calendar, so it can complete a job rather than take a message.
4. Run it for two weeks and read every transcript. Not a sample: every one.
5. Only then widen the scope. Agents fail by being given too much too early, not by being incapable.