Latency engineering
Voice AI latency is the silence a caller hears right before they hang up.
Every voice agent has a gap between the last word a caller says and the first sound it sends back. Zumu closes that gap with eleven separate engineering decisions, one processing stage at a time, not with a benchmark number.
- Endpoint detection through playout
- Tuned against real call incidents
- Warm caches, not cold starts
- A rule-based backstop for genuine silence
The anatomy of a turn
Five stages stand between the caller's last word and the agent's first sound
Everything below happens inside a second or two, every time a caller finishes a sentence. Eleven of Zumu's engineering decisions attach to one of these five stages. None of them is a single button. Together, they are why the wait feels shorter than it used to.
Endpoint detection
guess from a fixed pausea trained detector, not a timer
STT final
No optimization spent here
already fast enough to leave alone
LLM first token
a cold cache on the first callwarm before the caller finishes talking
TTS first byte
wait for the whole sentencespeak as soon as there is enough to say
Playout
silence while a tool call runsa caller who still believes someone is there
Why the wait matters
Two seconds of silence is not neutral
A caller does not read a pause as the agent thinking. They read it as three specific things, and none of them are patient.
They repeat themselves
in case the line dropped, or the agent never heard them the first time.
They talk over the agent
the moment it starts speaking again, because they already decided it needed a nudge.
They hang up
and call back, ask for a human, or try a competitor before the agent has said anything wrong at all.
The second one is why interruption detection matters as much as any of the eleven optimizations on this page. When a caller does talk over the agent, the system has to tell a genuine interruption from a stray cough or a passing truck almost instantly. Measured in production, Zumu's detector does that with 86% precision, so the agent stops talking because the caller started, not because a sound on the line got mistaken for one.
The eleven decisions
Where each one attaches
Grouped by the stage it shortens, in the order a call actually moves through them. A few make the pipe faster. A couple do not touch the clock at all, and say so.
Endpoint detection
Worker prewarm
Silero VAD and the turn detector load once when a worker process starts, not on every call. A caller never pays for that boot time. By the time they are connected, the listening pipeline is already warm.
Turn detection tuning
Endpoint detection runs on Silero VAD plus a trained turn detector, not a fixed pause timer. The thresholds were tuned against real incidents, not a benchmark, after slow or disfluent callers were cut off before they'd finished a sentence.
AEC warmup
The agent gives its own audio path a moment to settle before it starts listening for real. Skip that step and the agent hears its own voice come back through the line and mistakes it for the caller talking.
STT final
No numbered decision attaches here. Streaming transcription is already about as fast as speech itself, so nothing on this list was worth trading against it.
LLM first token
Connection pooling
Every worker keeps one shared HTTP client and one shared model client instead of opening a fresh connection per call. At the start of a busy shift, that is the difference between one warm handshake and hundreds of cold ones competing at once.
Prompt-cache warm ping
The instant a session starts, a tiny shadow request fires to the model provider before the caller has said a word, seeding the cache while the greeting plays. The first real turn of the call lands on a warm cache instead of a cold one.
Prompt-cache hardening
Every agent keeps a stable cache key tied to its exact prompt version, held warm for a full day. A large system prompt only has to be processed in full once. Every call after that reuses it instead of paying for it again.
Preemptive generation, kept off
The model could start drafting a reply before the caller has technically finished talking. Zumu turns that off on purpose. It reads well in a demo and badly on a caller who talks slowly or interrupts themselves, exactly who calls a phone line instead of opening an app.
TTS first byte
TTS time-to-first-byte tuning
The voice engine is told to speak in small chunks at first and larger ones once it is rolling, so the first sound leaves the pipeline as soon as there is enough of a sentence to say, not once the whole sentence is ready.
Pre-rendered greeting cache
The opening line an agent says is rendered ahead of time and pulled from storage instead of synthesized live. On a cache hit, greeting time-to-first-byte drops from roughly 230 to 900 milliseconds of live synthesis down to about 30 milliseconds, with a live fallback if the cache ever misses.
Playout
Background audio as latency masking
None of this makes a real tool call finish faster. While the agent is actually working, faint office ambience starts and a short verbal filler follows, so a caller hears a person still there instead of dead air while a lookup runs.
Dead-air rescue
The backstop for everything above. A rule-based state machine watches for genuine mutual silence, not just a pause, and works through one extension, then a nudge, then an emergency transfer, so a stuck call never just goes quiet on a caller.
The cache story
The fastest turn is the one the agent already prepared for
Two of the eleven decisions on this page are really one idea, applied twice: do not make the model start cold.
The greeting, on a cache hit
~30ms
cached greeting, on a hit
Live synthesis runs roughly 230 to 900 milliseconds when the cache misses. The live path stays wired in as the fallback, and a miss regenerates the cache for next time.
4.5s → 1.5–2.5s
first LLM call of a session, before and after the warm ping
A tiny shadow request seeds the prompt cache the moment a session starts, while the greeting is still playing.
0.98–0.99
prompt-cache hit ratio, measured across production traffic
A stable cache key tied to each agent’s exact prompt version stays warm for a full day, so a call hours after the last one still lands warm.
The best latency work is the two seconds nobody notices you saved.
Where this fits
Latency is one layer, not the whole story
- The platform
The system this pipeline runs inside
Latency is one layer of a larger architecture: five independently deployed services sharing one RLS-scoped database, built so a slow layer never has to become everyone’s problem.
See the architecture - The product
The agent this pipeline serves
Every stage on this page exists to serve one thing: an agent that answers, listens, and hands off like it was configured to, not like it is fighting its own plumbing.
See voice agents
Book a demo
Bring a call. We will show you where the seconds went.
Point Zumu at a real recording you have, one with a transfer if you have it, and we will walk through which of the eleven decisions above is doing the work on your own calls.
- No credit card for the demo
- Bring your own recording