Skip to main content

What are you looking for?

Explore our services and discover how we can help you achieve your goals

Client Workspace

Voice AI Agents: Speech-to-Text, LLM and Text-to-Speech Latency Budget

A voice AI agent feels natural when the silence between the caller's last word and the agent's first audio is short and predictable, so every stage needs its own target. Netbase splits that silence into endpointing, speech-to-text, model and speech-synthesis budgets, streams every stage and designs turn-taking before choosing vendors.

Book a solution review See the related solution

By Netbase AI and Automation Engineering Desk · Reviewed by David (CEO) · Updated · 11 min read

star

This guide is for CTOs, AI leads and contact-centre leads planning a phone or in-app voice agent, and is the voice companion to our generative AI development service.

In this guide

Why does a voice agent feel slow when every component is fast?

A voice agent feels slow because the caller hears the total silence after they stop talking, not component speed. The largest wait is often deciding that the caller has finished.

People are fast at this. A review of turn-taking research by Levinson and Torreira reports that gaps between speakers are of the order of 200 ms, while planning an utterance takes over 600 ms, so listeners predict the end of a turn and start planning early. A voice agent that waits for silence and only then starts every stage cannot match that rhythm.

Vendor figures differ widely. A caller-side benchmark published by Telnyx, measured from the end of the caller's speech in the recording to the first agent audio over the phone, reports platform medians between 1,296 and 1,740 ms. Measure your own call path, end to end, at the caller.

What goes into the latency budget?

The budget is the time from the caller's last word to the first audio they hear. Write it down as stages, give each stage a target at the 95th percentile, not the average, and measure each one separately so a regression points to its owner.

  1. Endpointing (turn detection)

    The system decides the caller has finished

    Main lever
    Turn-detection method and its timeout
    Typical failure
    Waits too long (dead air) or cuts in (interruptions)
  2. Speech-to-text finalisation

    The last words become a final transcript

    Main lever
    Streaming recognition with partial results
    Typical failure
    Final transcript arrives late or changes the meaning
  3. Retrieval and tool calls

    Data the answer needs is fetched

    Main lever
    Prefetch, read-only fast paths, async callbacks
    Typical failure
    A slow system call blocks the live turn
  4. Model first token

    The language model starts the reply

    Main lever
    Model size, prompt length, streaming
    Typical failure
    Long system prompts and large models on simple turns
  5. Text-to-speech first audio

    Synthesis starts speaking the first phrase

    Main lever
    Streaming synthesis by sentence or phrase
    Typical failure
    Waiting for the full reply before speaking
  6. Network and playout

    Audio travels to the caller and plays

    Main lever
    Region choice, codec, phone or app path
    Typical failure
    Extra hops through telephony or distant regions

Retell's own description of a streamed voice pipeline notes that streaming recognition runs in parallel with the caller's speech; endpointing, the model's first token and tool calls are where the budget is usually lost.

How to fill the budget template

Set the total from the call flow, give every stage a share, keep a reserve for tool calls, and treat any stage over its share at the 95th percentile as a defect. If the total cannot be met, change the design, not only the vendor: say a short holding phrase while a slow lookup runs, or move the lookup out of the live turn.

Which turn-taking method fits which call?

Turn-taking decides when the agent speaks and what happens when the caller talks over it. Each method fails in a different direction, so choose by which failure the call can tolerate.

Method How it decides When it fails Fits
Fixed silence timeout A set pause after speech ends Cuts in on slow speakers or waits on fast ones Short, scripted prompts such as yes or no
Voice activity detection Acoustic speech and silence Background noise, hesitation and "um" pauses Quiet channels and low-latency needs
Semantic end-of-turn model The words said so far, plus acoustics Unusual phrasing, other languages, model drift Open questions and natural conversation
Push-to-talk or manual commit The user or the app marks the turn Users forget to release or press In-app tools, noisy sites, dictation

OpenAI's turn-detection guide separates silence-based server detection from a semantic classifier that judges whether the user has finished from the words spoken, with an eagerness setting that trades waiting against cutting in. LiveKit's turn-handling documentation lists voice activity detection, speech-to-text endpointing, a turn-detector model, realtime-model detection and manual control, and pauses the agent's speech when the user starts talking.

Barge-in needs its own rules. Decide which prompts may be interrupted (legal disclosures often may not), whether "mm-hm" counts as an interruption, and what happens to a generated reply not yet spoken.

Cascaded pipeline or speech-to-speech model?

Component architecture sets the ceiling for both latency and control.

Criterion Cascaded (speech-to-text, LLM, text-to-speech) Speech-to-speech model Hybrid (speech-to-speech with text checks)
Control and audit Highest: text exists at every stage and can be checked or logged Lower: reasoning and speech happen in one session Medium: transcripts and tool results checked outside the model
Policy checks before speaking Easy: check the text reply before synthesis Harder: checks run alongside or after Possible on tool calls and selected turns
Component choice Each stage can be replaced or routed separately One provider for the whole turn One live model, other models for back-office steps
Latency shape Sum of streamed stages; needs careful overlap Fewer hand-offs between stages Close to speech-to-speech on simple turns
Choose it when Regulated wording, many integrations, mixed languages Open conversation where tone matters and risk is low A natural-sounding front line with governed actions

OpenAI's voice agent guide frames the same choice: a speech-to-speech session when one model should interpret audio, decide and respond in speech, and a chained pipeline when each stage needs to be visible or replaceable, for example to store the transcript and run policy checks before the agent answers.

How to build a voice agent against a latency budget

  1. Map the call flow before choosing vendors

    List intents, the systems each touches and which turns need a person; a voice agent is a tool-using agent under a strict clock.

  2. Set the total and the stage budget

    Agree the percentile targets with the business owner of the call.

  3. Choose the architecture and turn-taking method

    Use the two tables above; pick the failure the call can tolerate.

  4. Stream every stage

    Partial transcripts, streamed model tokens and phrase-by-phrase synthesis, with nothing in between that buffers a whole reply.

  5. Keep slow work out of the live turn

    Prefetch customer context when the call connects, keep retrieval small and fast, and move writes to an asynchronous step confirmed later.

  6. Use the smallest model that passes the evaluation

    Routing and slot filling often fit a small language model; keep the larger model for the turns that need it.

  7. Measure at the caller, then tune

    Timestamp each stage on recorded calls and review the slowest 5 percent of turns and every interruption weekly.

When should you not build a voice agent?

A voice agent is the wrong first step when the answers are long, the wording is regulated and must be reviewed before it is said, or the channel is too noisy for reliable recognition. In those cases a text channel with hand-off, or a voice menu that only routes calls to people, carries less risk.

It is also premature when the systems behind the call cannot answer a lookup within a fraction of the budget.

Where AI fits

AI handles recognition, end-of-turn prediction, the reply and synthesis. Engineers still set the budget, choose the turn-taking rules, write the tool contracts, review recorded calls and decide which turns go to a person. Netbase works with commercial and open-weight models from OpenAI, Anthropic (Claude), Google (Gemini) and Meta (Llama), among others, chosen per project. Each item below states how mature it is at Netbase.

What delivery record exists, and what does not

  • What exists. Netbase built a WhatsApp AI chatbot that combines OpenAI language models with an intent engine and a CRM sync for a client that is not named, and has delivered retrieval-augmented knowledge assistants, document AI and MLOps pipelines for clients that are not named. The named delivered AI feature is 4over4's recommendation engine, which suggests products from browsing history and previous purchases. Most Netbase projects are delivered on fixed-price contracts.
  • What does not. No published Netbase case covers a voice agent, and none reports a response time, interruption rate, containment rate or call volume. The budget template, turn-taking matrix and build order are method guidance drawn from the cited sources and engineering practice.

What are the alternatives to a custom voice agent?

  • A voice platform's hosted agent. Choose it when the call flow is simple, the integrations are few and the platform's own turn-taking and telephony fit.
  • A text agent with a call-back or hand-off. Choose it when answers are long or need review; the customer service AI route covers text replies with hand-off.
  • A knowledge assistant for the human agents. Choose it when accuracy matters more than automation; the enterprise knowledge and RAG service covers that path, and the human stays the voice.

What are the limits of this guide?

  • Vendor latency figures quoted here are each vendor's own measurements on its own stack; they show the shape of the problem, not a target for yours.
  • Turn-taking research and the detection methods named here were described mostly for English; test recognition and end-of-turn behaviour in every language the agent will hear.

Plan the next step with a Netbase consultant

Frequently asked questions

Aim for a total gap that suits the call flow and hold it at the 95th percentile, measured from the caller's last word to the first audio they hear. A predictable gap matters as much as a short one.

Deciding that the caller has finished speaking is often the largest and least visible wait, followed by the language model's first token and any tool call inside the turn. Streaming recognition mostly runs while the caller is still talking, so it rarely dominates.

Use a pipeline when wording must be checked, logged or kept in a regulated form, or when each stage must be replaceable. Use a speech-to-speech model for open conversation where tone matters and the risk of a wrong sentence is low.

Decide per prompt whether the caller may interrupt, ignore short acknowledgements such as "mm-hm", stop speaking immediately on a real interruption and discard the unspoken part of the reply.

Next step

Send us the call flow, the systems the agent must read and a few recorded calls, and we will book a solution review to set the latency budget and the turn-taking design before any vendor is chosen.

Generative AI development with evaluation and human review Generative AI development with evaluation and human review

Netbase builds generative AI features for product and operations teams: drafting, summarising, rewriting and content generation inside the software your people and customers already use. Every feature ships with an evaluation set, a human review step where output matters and limits on cost. Generative AI is a capability we are growing, and we label its maturity honestly.

Learn More
line
Enterprise AI knowledge assistants: RAG engineering with access control Enterprise AI knowledge assistants: RAG engineering with access control

Netbase engineers retrieval-augmented generation (RAG) over company knowledge: the ingestion, search, permission checks, cited answers and evaluation that let an internal assistant or a website RAG chatbot answer from your own documents. It is a growth capability with access control designed in from the first prototype, and Netbase has delivered a RAG knowledge assistant for an unnamed client.

Learn More
line
AI customer service automation connected to your orders and CRM, with human hand-off AI customer service automation connected to your orders and CRM, with human hand-off

AI customer service automation is a support layer where AI chat answers routine questions and triages tickets using live order and CRM data, then hands anything sensitive or uncertain to a person with the full context. Netbase builds it as a growth capability, starting from its AI chatbot and WorkChat integrator module.

Learn More
line

Work with Netbase remotely

Every Netbase engagement is delivered remotely, so clients in any country and any industry can work with us; Netbase has worked with clients in 80 countries. Contracts and invoices are issued by Netbase JSC, and payment is made by bank transfer to Netbase JSC's company account or PayPal Business. Talk to the CEO, Huy Nguyen (David), on WhatsApp or by email.

Contact Netbase

Discuss a project

Netbase JSC helps organizations design, build, modernize, and operate digital products and AI-enabled business systems.
Project enquiries

project@netbasejsc.com

WhatsApp

+84 937 869 689

Office address

91 Nguyen Chi Thanh, Dong Da, Hanoi, Vietnam

Get in touch

Tell us what you want to build, modernize, or operate.

Tell us what you want to build, modernize, or operate.

Contact Netbase