How AI Voice Agents Work

The four parts of a call

1. Speech to text. The caller's audio is transcribed continuously, not after they finish, which is what lets the agent be interrupted.
2. The model. The transcript plus your system instructions, your business information and the conversation so far are given to a language model, which decides what to say or do next.
3. Tools and actions. The model can call functions you have given it: check calendar availability, create a booking, look up a customer, send an SMS, transfer the call.
4. Text to speech. The reply is spoken with a synthetic voice, streamed as it is generated so there is no long pause.

Why latency is the whole game

Humans notice a conversational gap of about half a second. Every part of the loop adds delay, so voice platforms work hard to overlap them — starting to transcribe before the caller stops, starting to speak before the full reply is generated. When you hear a voice agent that feels stilted, latency is usually the reason rather than the words themselves.

This is also why voice agents are architected differently from chatbots. A chat interface can afford two seconds of thinking. A phone call cannot.

What the instructions contain

The system instructions are where the behaviour lives, and they are more detailed than people expect. A working agent's instructions cover: who it is and which business it represents, what it may and may not say, the questions it must ask, how to handle someone who wants a human, what to do when it does not know, how to end a call, and how to behave if the caller is upset.

Your business information — services, hours, pricing approach, policies, locations — is supplied separately so it can be updated without rewriting the behaviour.

Where it goes wrong

• Names, street addresses and email addresses are the hardest things to capture accurately by voice. Good agents confirm them back and spell them out.
• Heavy background noise degrades transcription badly.
• Very fast speech, strong accents and code-switching between languages all raise error rates.
• If the instructions are vague, the model fills gaps with plausible-sounding invention. This is the single most important thing to guard against.
• Tool failures — a calendar that does not respond — need explicit handling, or the agent will improvise around them.

How to test one properly

1. Call it yourself, ten times, with your ten most common caller situations.
2. Try to break it: interrupt, change your mind, ask something out of scope, ask for a human.
3. Check that everything it booked actually landed in the calendar correctly.
4. Read the transcripts, not just the summaries.
5. Have someone with a strong accent, and someone speaking Spanish, call it too.