Building an AI tele-calling agent that completes the task
Every business we work with has the same silent leak. Somebody calls the veterinary clinic at 7:40pm to move an appointment, hears six rings and a voicemail greeting, hangs up, and books with whoever answers next. Nobody logs it. It never shows on a report. At the end of the quarter the owner sees revenue slightly below target and blames the market.
An AI tele-calling agent is the cheapest fix for that leak, and also the easiest thing in modern software to build badly. Demos are trivial: wire speech recognition to a language model, pipe the reply through a text-to-speech engine, and you have something that talks. Putting that on a real phone line, in a room with a dishwasher running, in front of a caller with an accent and a toddler, is a different discipline entirely.
This is how we build them, what breaks, and the numbers we hold ourselves to.
The latency budget is the architecture
Humans tolerate about 800 milliseconds of silence in conversation before they assume something is wrong. They start repeating themselves, they say "hello?", they hang up. Every architectural decision in a voice agent falls out of that single number, so we write the budget down before we write any code.
| Stage | Budget | What blows it |
|---|---|---|
| Endpointing | 120 ms | Waiting for silence instead of predicting turn-end |
| Speech to text | 200 ms | Batch transcription rather than streaming partials |
| Reasoning | 300 ms | Oversized prompts, sequential tool calls, no streaming |
| Text to speech | 180 ms | Generating the whole sentence before playback starts |
| Transport | 100 ms | Region hops between telephony and inference |
The trick that buys most of the headroom is refusing to work in sequence. Transcription streams partial words while the caller is still speaking. The model starts drafting on those partials. Speech synthesis begins on the first clause while the rest is still being generated. By the time the caller stops talking, the first syllable of the answer is already on the wire.
Barge-in, or how to sound human
The single feature that separates a convincing agent from an obvious robot is barge-in: the caller interrupts, and the agent stops mid-word and listens. It sounds small. It requires full duplex audio, echo cancellation so the agent does not transcribe its own voice, and the discipline to discard the half-spoken response and re-plan from what the caller actually said.
Get it wrong and you produce the worst experience in the category — a system that talks over a frustrated human. We treat interrupt latency as a hard requirement and measure it in every release.
Answering is not the job. Completing is.
A voice agent that takes a message has moved work, not removed it. Somebody still opens the inbox at nine the next morning and calls back. Our threshold for shipping is that the agent finishes the task inside the call and writes it back to the system of record before the line drops.
- Booking: reads live availability, holds the slot during the conversation, confirms it, releases the hold if the caller changes their mind.
- Rescheduling and cancellation: identifies the caller, applies your notice policy, frees the slot for the waiting list automatically.
- Outbound reminders and recalls: vaccination due, six-month dental check, table confirmation the afternoon before, invoice past thirty days.
- Qualification: captures what the enquiry is about and routes it to the right person with a summary already attached.
- Escalation: hands to a human on request, on repeated confusion, or on any word that suggests an emergency.
Escalation is a safety feature
In healthcare-adjacent settings this is not optional. A veterinary line gets calls about a dog that has eaten chocolate. A dental line gets facial swelling. The agent needs a hard-coded triage layer that recognises urgency vocabulary and transfers immediately, before any model reasoning happens. We build that as a deterministic rule set, not as a prompt instruction, because prompts drift and rules do not.
What it costs to run, honestly
Per-minute cost is dominated by inference, not telephony. Self-hosting the speech and language stack lowers marginal cost significantly at volume and raises operational burden — you now own GPU capacity planning and model updates. Managed APIs invert that trade. For most clients under roughly two hundred calls a day, managed wins; above it, a hybrid does. We model both before recommending either, and we show the working.
How we roll it out
Never as a big-bang replacement of a phone line. Week one, the agent takes overflow only — calls that would have hit voicemail anyway, where the alternative is nothing. Week three, it takes after-hours in full. Week six, it fronts the line during business hours with instant transfer available. Every stage is measured against the previous one, and the rollback is a single configuration change.
The result our clients notice first is not the technology. It is that the front desk stops flinching when the phone rings during a consultation.
Accents, noise and names nobody spells the same way
Generic speech recognition is excellent at generic English and mediocre at the words your business actually uses. A veterinary line needs to hear "Frontline", "Bravecto" and a spaniel called Nyx. A dental line needs "periodontitis" and a surname the caller has been spelling out for people their entire life. A restaurant needs the name of a dish that exists in no dictionary.
We handle this with vocabulary biasing rather than hope: your service list, staff names, product catalogue and top two thousand customer surnames are loaded as recognition hints, and confidence thresholds are set per field. Where confidence drops below the threshold on something expensive to get wrong — a date, a phone number, a dosage — the agent confirms explicitly instead of guessing. Digits are grouped the way humans say them. Spellings are read back in the phonetic alphabet when the caller is clearly spelling.
Background noise is the other half of the problem. Kitchens, waiting rooms and car speakerphones are not test conditions, they are the normal case, so we evaluate against recordings from the client's own line rather than clean studio samples. A model that scores well on clean audio and badly at 40 decibels of ambient clatter is a model that fails on a Friday evening.
What we report every week after go-live
An agent that nobody measures drifts quietly. These are the six numbers we put in front of clients, and the thresholds at which we act on them.
- Containment rate: the share of calls resolved without a human. Useful, but meaningless alone — an agent can contain a call by frustrating someone into hanging up.
- Task completion by intent: booked, rescheduled, cancelled, answered, qualified. Reported separately, because averages hide the intent that is broken.
- Transfer rate and reason: requested, repeated confusion, out of scope, urgency detected. Rising confusion transfers means the model needs work; rising out-of-scope means the scope was drawn too tightly.
- Recognition accuracy on your vocabulary: not the vendor's published benchmark. Yours, on your terms, sampled weekly.
- Time to first word: the latency the caller actually experiences, at the ninety-fifth percentile, not the average.
- Abandonment inside the first ten seconds: the clearest signal that people do not want to talk to it.
Recording, consent and what happens to the audio
Voice systems collect some of the most sensitive data a business holds. In healthcare-adjacent settings, a call recording can contain clinical detail; in any setting it contains names, numbers and payment discussion. Consent announcements need to match the jurisdiction, and "this call may be recorded" is not a substitute for a lawful basis.
We default to a conservative posture: audio retained only as long as the client's policy requires, transcripts redacted for card numbers and identifiers before they reach any log, storage in the client's region, and a documented deletion path that someone can actually execute when a caller asks. If a client wants a shorter retention window than we propose, we shorten it. Nobody has ever regretted holding less voice data.
What an AI tele-calling agent should not do
It should not diagnose. It should not negotiate a bill. It should not attempt to retain an angry customer. It should not pretend to be human when asked directly — we answer that question honestly every time, and in three years of deployments that honesty has never once cost a booking.
Drawing those lines narrowly at the start is what makes the rest of the system trustworthy. An agent that does five things reliably is worth more than one that attempts thirty and surprises you on a Sunday.
Frequently asked
How long does it take to deploy an AI tele-calling agent?
Four to six weeks for a single-site deployment covering bookings, rescheduling and enquiries, including integration with your calendar or practice management system. Multi-site or complex telephony estates take longer, usually eight to ten weeks.
Will it work with our existing phone number and system?
Yes. The agent sits in front of your existing number through your telephony provider, so the number your customers know does not change. Transfers hand back into your normal extensions.
What happens if the agent cannot handle a call?
It transfers to a human with a summary of the conversation already attached, so nobody has to ask the caller to start again. Escalation triggers include explicit requests, repeated confusion, and any urgency vocabulary we define with you upfront.
Which businesses get the most from it?
Anyone losing calls outside business hours or during busy periods — veterinary and dental clinics, hospitals, restaurant groups and travel agencies see the fastest payback, because a missed call there is a directly measurable lost booking.