Voice agents sound simple in theory: audio in, intelligent response out. In practice, builders face a bewildering choice of vendors, each handling a different piece of the puzzle. A new technical guide from Telnyx breaks down the voice-agent stack into six distinct layers, revealing why the architecture decision you make first cascades through every tool choice that follows.

The core problem is that no single vendor dominates every layer. Speech-to-text (STT) models, text-to-speech (TTS) systems, language models, orchestration platforms, testing tools, and telephony infrastructure are all separate domains with separate leaders. Choosing the right combination requires understanding not just which tools are best, but which architecture fits your needs.

What Are the Two Main Paths for Building Voice Agents?

Builders face a fundamental choice: modular or end-to-end. The modular approach chains separate tools together. Audio enters through a telephony layer, a speech-to-text model converts it to text, a language model (LLM) decides what to say, and a text-to-speech model speaks the response. An orchestration layer coordinates the timing, interruptions, tool calls, and state throughout the loop.

The payoff of modularity is control. Developers can read the intermediate transcripts, swap any layer independently, and debug the exact stage where something broke. But this flexibility comes with complexity: more moving parts, more vendors to manage, and more potential failure points.

The alternative is a native speech-to-speech model that receives audio and returns audio directly. One model handles the entire conversation without separate STT, LLM, or TTS stages. This approach preserves more acoustic context and eliminates intermediate steps, but trades pipeline visibility for simplicity. Developers get less insight into what the model actually understood and less freedom to substitute components.

How to Choose the Right Tools for Each Layer?

• Speech-to-Text Decision Factor: Entity accuracy and turn detection matter most. Misheard names cause wrong lookups, dropped digits trigger wrong account pulls, and late end-of-turn signals make agents talk over callers. AssemblyAI's Universal-3.5 Pro Realtime posts a 6.99% word error rate and 15.31% entity error rate on voice-agent benchmarks, handling code-switching across 18 languages with live speaker identification.
• Text-to-Speech Decision Factor: Initial audio quality, content accuracy, and naturalness determine whether callers perceive the agent as trustworthy. The choice here affects how natural the voice sounds and how well it handles varied content without artifacts or mispronunciations.
• Native Speech-to-Speech Decision Factor: Conversation quality and tool use capability matter when you choose a single model to handle the entire audio loop. OpenAI's GPT-Realtime-2.1, Google's Gemini, and xAI's models compete on how well they maintain context and execute backend actions.
• Orchestration Decision Factor: Control, hosting options, and operational scope determine whether you manage the coordination layer yourself or delegate it to a platform. This choice affects how much visibility you have into the agent's behavior and how easily you can modify it.
• Testing Decision Factor: Task completion and regression coverage ensure the agent actually solves the problem it was built for. Testing tools measure whether the agent understood the caller, completed the task, performed the correct backend action, and responded within acceptable time limits.
• Telephony Decision Factor: Call control, media streaming, and routing capabilities determine how the agent connects to real phone lines and handles inbound or outbound calls. This layer is where audio physically enters and exits the system.

The stakes are concrete and measurable. A voice agent's transcript has exactly one reader: the language model. Every action the agent takes flows from what the model believes was said. This means the dimensions that matter are the ones that decide those actions: entity accuracy on addresses, account numbers, phone numbers, and names; streaming stability under background noise; end-of-turn detection; code-switching between languages; speaker handling; and response timing.

Not every stack needs a separate vendor at every layer. Managed platforms combine several layers into one product, which can be exactly right for teams that want someone else's support team to handle operations. The real question is where you want control and where you want to outsource responsibility. Some teams need to read intermediate transcripts and debug at the text level. Others prefer a simpler, more opaque system that just works.

The guide evaluates tools based on independent benchmarks, current product documentation, published pricing, and claimed production capabilities, with benchmarks current as of August 2026. Each category is judged on criteria specific to its layer: for speed, real-time fit and latency matter; for output quality, accuracy and task completion drive the decision; for operations, deployment controls, integration options, and actual cost determine feasibility.

The voice-agent market is maturing fast, but the tooling landscape remains fragmented. Builders who understand the six-layer stack and the architectural choice at its core can navigate the options more confidently. Those who skip this analysis often end up rebuilding their stack halfway through deployment, discovering too late that their tool choices don't work together the way they expected.