Voice AI systems are being optimized for the wrong speed metric, causing developers to build agents that feel sluggish even when they technically respond quickly. A comprehensive benchmark of voice inference APIs reveals that time-to-first-token (TTFT), the standard measurement used to compare AI providers, tells only half the story for voice applications. For natural-sounding conversations, what matters is how fast the AI completes its first full sentence, not just when it starts generating text.

What's the Difference Between TTFT and What Users Actually Experience?

The confusion stems from a mechanical limitation of text-to-speech technology. A text-to-speech model cannot produce audio from partial words or fragments. It needs a complete clause or sentence before it can synthesize sound. This means that even if an AI language model generates its first token in 100 milliseconds, the user hears nothing until enough tokens arrive to form a complete phrase.

LiveKit, a platform for building voice applications, introduced a more accurate metric called time-to-first-sentence (TTFS) to measure what users actually perceive. The distinction matters because a provider that wins on TTFT but loses on token generation speed will still feel slow to end users. For example, a model that produces its first token in 200 milliseconds but then generates tokens slowly will leave users waiting longer than a model with a slightly higher TTFT but faster overall output.

How Much Latency Can a Conversation Actually Tolerate?

Researchers have established practical targets for natural-sounding voice interactions. Typical human response time in conversation is around 500 milliseconds. Pauses beyond 800 milliseconds start to feel unnatural and awkward. For voice-to-voice latency, experts recommend targeting 800 milliseconds as a median, with 1,500 milliseconds acceptable for proof-of-concept demonstrations.

This latency budget must be divided across multiple stages of processing. Each component of the voice stack consumes milliseconds that users can hear. The breakdown looks roughly like this:

• Speech-to-text conversion: Approximately 100 to 200 milliseconds to transcribe what the user said
• Language model inference: Approximately 300 to 500 milliseconds for the AI to generate a response
• Text-to-speech synthesis: Approximately 100 to 200 milliseconds to convert text to audio
• Network transmission: Approximately 50 to 150 milliseconds for data to travel between servers and the user

These stages run sequentially, not in parallel, which means the total time adds up quickly. A voice agent that exceeds 1.2 seconds end-to-end will feel noticeably delayed to users.

Why Different Providers Show Wildly Different Speed Numbers

When comparing inference API providers, the same model can show dramatically different latency depending on where it runs. For instance, OpenAI's GPT-5.6 Luna model measured 0.59 seconds on Amazon Bedrock but 0.74 seconds on OpenAI's own API. The difference comes down to hosting infrastructure, routing decisions, and how efficiently each provider packs requests onto their hardware.

LiveKit's own inference service demonstrates how architectural choices affect real-world performance. Running Gemma 4 31B with speculative decoding and deliberately under-packed GPUs to minimize queueing delays, LiveKit achieved 192 milliseconds TTFT for that model. The same model via OpenRouter measured 1,876 milliseconds, nearly 10 times slower. The tradeoff is cost; LiveKit's approach runs at approximately $1.20 per million output tokens.

Benchmark methodology also shapes the results. Most providers now test with 10,000-token input prompts rather than the smaller 1,000-token prompts used previously. Longer prompts raise both TTFT and output speed, but they better reflect real production voice agents that front-load policy instructions, persona details, escalation rules, and retrieved data.

How to Evaluate Voice AI Performance Beyond the Headline Number

• Measure time-to-first-sentence, not just first token: Request TTFS metrics from providers, which account for how long it takes to complete the first audible phrase. This reflects what users actually experience in conversation.
• Test with realistic prompt sizes: Benchmark providers using 10,000-token input prompts that include system instructions, context, and tool definitions, not minimal test prompts that don't represent production workloads.
• Account for server location and network latency: TTFT includes network delay, which varies based on geographic location. Test from the same region where your application will run, or request latency breakdowns from your provider.
• Verify tokens-per-second output speed: A fast first token means nothing if the model then generates subsequent tokens slowly. Request output speed metrics alongside TTFT to calculate how long the first sentence actually takes to complete.
• Check for consistency across runs: TTFT varies substantially between benchmark runs as providers adjust inference stacks and model weights. Ask for median latency over multiple hours or days, not single-run measurements.

The practical target for voice applications is approximately 700 milliseconds of TTFT budget for the language model component alone, leaving room for speech-to-text, text-to-speech, and network delays within the 1,200-millisecond total latency ceiling.

Which Providers Actually Meet the Speed Requirements?

Cerebras and Groq stand out for combining respectable TTFT with exceptional token generation speed. Cerebras's gpt-oss-120b model generates 1,697 tokens per second, meaning sentences complete almost immediately after the first token arrives. This combination is particularly strong for TTFS, the metric that matters for voice.

Smaller, faster models like Gemma 4 31B offer a different tradeoff. On LiveKit's infrastructure, Gemma 4 31B achieved 354 milliseconds time-to-first-sentence across full conversations, compared to 1,034 milliseconds for Gemini 2.5 Flash and 1,088 milliseconds for GPT-4.1. Gemma 4 31B scored 75.6% on IFBench, nearly matching GPT-5.5's 75.9%, suggesting that smaller models can deliver both speed and capability for voice applications.

Speech-to-text latency also matters. Deepgram Flux achieves approximately 260 milliseconds end-of-turn detection at default settings, while Cartesia Ink-2 reports 100 milliseconds transcript latency. These differences directly impact how quickly the system knows the user has finished speaking and can begin generating a response.

The takeaway for developers building voice agents is clear: stop optimizing for TTFT alone. Measure time-to-first-sentence, account for the full latency budget across all components, and test with realistic production workloads. The providers that win on paper may not deliver the conversational experience users expect.