This guide is for CTOs, AI leads and contact-centre leads planning a phone or in-app voice agent, and is the voice companion to our generative AI development service.
In this guide
- Why does a voice agent feel slow when every component is fast?
- What goes into the latency budget?
- Which turn-taking method fits which call?
- Cascaded pipeline or speech-to-speech model?
- How to build a voice agent against a latency budget
- When should you not build a voice agent?
- Where AI fits
- What delivery record exists, and what does not
- What are the alternatives to a custom voice agent?
- What are the limits of this guide?
- Frequently asked questions
- Next step
Why does a voice agent feel slow when every component is fast?
A voice agent feels slow because the caller hears the total silence after they stop talking, not component speed. The largest wait is often deciding that the caller has finished.
People are fast at this. A review of turn-taking research by Levinson and Torreira reports that gaps between speakers are of the order of 200 ms, while planning an utterance takes over 600 ms, so listeners predict the end of a turn and start planning early. A voice agent that waits for silence and only then starts every stage cannot match that rhythm.
Vendor figures differ widely. A caller-side benchmark published by Telnyx, measured from the end of the caller's speech in the recording to the first agent audio over the phone, reports platform medians between 1,296 and 1,740 ms. Measure your own call path, end to end, at the caller.
What goes into the latency budget?
The budget is the time from the caller's last word to the first audio they hear. Write it down as stages, give each stage a target at the 95th percentile, not the average, and measure each one separately so a regression points to its owner.
-
Endpointing (turn detection)
The system decides the caller has finished
- Main lever
- Turn-detection method and its timeout
- Typical failure
- Waits too long (dead air) or cuts in (interruptions)
-
Speech-to-text finalisation
The last words become a final transcript
- Main lever
- Streaming recognition with partial results
- Typical failure
- Final transcript arrives late or changes the meaning
-
Retrieval and tool calls
Data the answer needs is fetched
- Main lever
- Prefetch, read-only fast paths, async callbacks
- Typical failure
- A slow system call blocks the live turn
-
Model first token
The language model starts the reply
- Main lever
- Model size, prompt length, streaming
- Typical failure
- Long system prompts and large models on simple turns
-
Text-to-speech first audio
Synthesis starts speaking the first phrase
- Main lever
- Streaming synthesis by sentence or phrase
- Typical failure
- Waiting for the full reply before speaking
-
Network and playout
Audio travels to the caller and plays
- Main lever
- Region choice, codec, phone or app path
- Typical failure
- Extra hops through telephony or distant regions
Retell's own description of a streamed voice pipeline notes that streaming recognition runs in parallel with the caller's speech; endpointing, the model's first token and tool calls are where the budget is usually lost.
How to fill the budget template
Set the total from the call flow, give every stage a share, keep a reserve for tool calls, and treat any stage over its share at the 95th percentile as a defect. If the total cannot be met, change the design, not only the vendor: say a short holding phrase while a slow lookup runs, or move the lookup out of the live turn.
Which turn-taking method fits which call?
Turn-taking decides when the agent speaks and what happens when the caller talks over it. Each method fails in a different direction, so choose by which failure the call can tolerate.
| Method | How it decides | When it fails | Fits |
|---|---|---|---|
| Fixed silence timeout | A set pause after speech ends | Cuts in on slow speakers or waits on fast ones | Short, scripted prompts such as yes or no |
| Voice activity detection | Acoustic speech and silence | Background noise, hesitation and "um" pauses | Quiet channels and low-latency needs |
| Semantic end-of-turn model | The words said so far, plus acoustics | Unusual phrasing, other languages, model drift | Open questions and natural conversation |
| Push-to-talk or manual commit | The user or the app marks the turn | Users forget to release or press | In-app tools, noisy sites, dictation |
OpenAI's turn-detection guide separates silence-based server detection from a semantic classifier that judges whether the user has finished from the words spoken, with an eagerness setting that trades waiting against cutting in. LiveKit's turn-handling documentation lists voice activity detection, speech-to-text endpointing, a turn-detector model, realtime-model detection and manual control, and pauses the agent's speech when the user starts talking.
Barge-in needs its own rules. Decide which prompts may be interrupted (legal disclosures often may not), whether "mm-hm" counts as an interruption, and what happens to a generated reply not yet spoken.
Cascaded pipeline or speech-to-speech model?
Component architecture sets the ceiling for both latency and control.
| Criterion | Cascaded (speech-to-text, LLM, text-to-speech) | Speech-to-speech model | Hybrid (speech-to-speech with text checks) |
|---|---|---|---|
| Control and audit | Highest: text exists at every stage and can be checked or logged | Lower: reasoning and speech happen in one session | Medium: transcripts and tool results checked outside the model |
| Policy checks before speaking | Easy: check the text reply before synthesis | Harder: checks run alongside or after | Possible on tool calls and selected turns |
| Component choice | Each stage can be replaced or routed separately | One provider for the whole turn | One live model, other models for back-office steps |
| Latency shape | Sum of streamed stages; needs careful overlap | Fewer hand-offs between stages | Close to speech-to-speech on simple turns |
| Choose it when | Regulated wording, many integrations, mixed languages | Open conversation where tone matters and risk is low | A natural-sounding front line with governed actions |
OpenAI's voice agent guide frames the same choice: a speech-to-speech session when one model should interpret audio, decide and respond in speech, and a chained pipeline when each stage needs to be visible or replaceable, for example to store the transcript and run policy checks before the agent answers.
How to build a voice agent against a latency budget
-
Map the call flow before choosing vendors
List intents, the systems each touches and which turns need a person; a voice agent is a tool-using agent under a strict clock.
-
Set the total and the stage budget
Agree the percentile targets with the business owner of the call.
-
Choose the architecture and turn-taking method
Use the two tables above; pick the failure the call can tolerate.
-
Stream every stage
Partial transcripts, streamed model tokens and phrase-by-phrase synthesis, with nothing in between that buffers a whole reply.
-
Keep slow work out of the live turn
Prefetch customer context when the call connects, keep retrieval small and fast, and move writes to an asynchronous step confirmed later.
-
Use the smallest model that passes the evaluation
Routing and slot filling often fit a small language model; keep the larger model for the turns that need it.
-
Measure at the caller, then tune
Timestamp each stage on recorded calls and review the slowest 5 percent of turns and every interruption weekly.
When should you not build a voice agent?
A voice agent is the wrong first step when the answers are long, the wording is regulated and must be reviewed before it is said, or the channel is too noisy for reliable recognition. In those cases a text channel with hand-off, or a voice menu that only routes calls to people, carries less risk.
It is also premature when the systems behind the call cannot answer a lookup within a fraction of the budget.
Where AI fits
AI handles recognition, end-of-turn prediction, the reply and synthesis. Engineers still set the budget, choose the turn-taking rules, write the tool contracts, review recorded calls and decide which turns go to a person. Netbase works with commercial and open-weight models from OpenAI, Anthropic (Claude), Google (Gemini) and Meta (Llama), among others, chosen per project. Each item below states how mature it is at Netbase.
- In delivered work
WhatsApp AI chatbot with CRM integration
A text chatbot using OpenAI language models with an intent engine and a CRM sync, for a client that is not named.
- In delivered work
RAG knowledge assistant
A retrieval-augmented assistant for a client that is not named, the knowledge layer a voice agent reads from.
- Available capability
Voice agents built to a latency budget
The budget, turn-taking matrix and build order on this page; not yet tied to a published case.
What delivery record exists, and what does not
- What exists. Netbase built a WhatsApp AI chatbot that combines OpenAI language models with an intent engine and a CRM sync for a client that is not named, and has delivered retrieval-augmented knowledge assistants, document AI and MLOps pipelines for clients that are not named. The named delivered AI feature is 4over4's recommendation engine, which suggests products from browsing history and previous purchases. Most Netbase projects are delivered on fixed-price contracts.
- What does not. No published Netbase case covers a voice agent, and none reports a response time, interruption rate, containment rate or call volume. The budget template, turn-taking matrix and build order are method guidance drawn from the cited sources and engineering practice.
What are the alternatives to a custom voice agent?
- A voice platform's hosted agent. Choose it when the call flow is simple, the integrations are few and the platform's own turn-taking and telephony fit.
- A text agent with a call-back or hand-off. Choose it when answers are long or need review; the customer service AI route covers text replies with hand-off.
- A knowledge assistant for the human agents. Choose it when accuracy matters more than automation; the enterprise knowledge and RAG service covers that path, and the human stays the voice.
What are the limits of this guide?
- Vendor latency figures quoted here are each vendor's own measurements on its own stack; they show the shape of the problem, not a target for yours.
- Turn-taking research and the detection methods named here were described mostly for English; test recognition and end-of-turn behaviour in every language the agent will hear.
Plan the next step with a Netbase consultant
Frequently asked questions
Aim for a total gap that suits the call flow and hold it at the 95th percentile, measured from the caller's last word to the first audio they hear. A predictable gap matters as much as a short one.
Deciding that the caller has finished speaking is often the largest and least visible wait, followed by the language model's first token and any tool call inside the turn. Streaming recognition mostly runs while the caller is still talking, so it rarely dominates.
Use a pipeline when wording must be checked, logged or kept in a regulated form, or when each stage must be replaceable. Use a speech-to-speech model for open conversation where tone matters and the risk of a wrong sentence is low.
Decide per prompt whether the caller may interrupt, ignore short acknowledgements such as "mm-hm", stop speaking immediately on a real interruption and discard the unspoken part of the reply.
Next step
Send us the call flow, the systems the agent must read and a few recorded calls, and we will book a solution review to set the latency budget and the turn-taking design before any vendor is chosen.
Related services and solutions
Generative AI development with evaluation and human review Netbase builds generative AI features for product and operations teams: drafting, summarising, rewriting and content generation inside the software your people and customers already use. Every feature ships with an evaluation set, a human review step where output matters and limits on cost. Generative AI is a capability we are growing, and we label its maturity honestly.
Learn More
Enterprise AI knowledge assistants: RAG engineering with access control Netbase engineers retrieval-augmented generation (RAG) over company knowledge: the ingestion, search, permission checks, cited answers and evaluation that let an internal assistant or a website RAG chatbot answer from your own documents. It is a growth capability with access control designed in from the first prototype, and Netbase has delivered a RAG knowledge assistant for an unnamed client.
Learn More
AI customer service automation connected to your orders and CRM, with human hand-off AI customer service automation is a support layer where AI chat answers routine questions and triages tickets using live order and CRM data, then hands anything sensitive or uncertain to a person with the full context. Netbase builds it as a growth capability, starting from its AI chatbot and WorkChat integrator module.
Learn More
Work with Netbase remotely
Every Netbase engagement is delivered remotely, so clients in any country and any industry can work with us; Netbase has worked with clients in 80 countries. Contracts and invoices are issued by Netbase JSC, and payment is made by bank transfer to Netbase JSC's company account or PayPal Business. Talk to the CEO, Huy Nguyen (David), on WhatsApp or by email.
Discuss a project
Netbase JSC helps organizations design, build, modernize, and operate digital products and AI-enabled business systems.project@netbasejsc.com
+84 937 869 689
91 Nguyen Chi Thanh, Dong Da, Hanoi, Vietnam
Get in touch
Tell us what you want to build, modernize, or operate.