Google is fundamentally rethinking how AI understands human speech, shifting from translating words to grasping emotion, tone, and cultural context in over 300 languages spoken by 7 billion people. Rather than converting audio to text and back again, the company is training models like Gemini to process sound directly, capturing the richness of how people actually talk: the hesitations, overlaps, code-switching between languages, and emotional nuance that traditional systems strip away.

Why Does AI Struggle With How People Really Talk?

For decades, speech recognition systems followed a rigid pipeline: transcribe audio into text, process that text, then synthesize it back into speech. This approach worked, but it lost everything that makes human communication meaningful. People don't speak in perfect, grammatical sentences. They laugh, interrupt themselves, mix languages mid-sentence like Spanglish or Hinglish, and layer emotion and intent into every word.

The old multi-step process required substantial specialized knowledge to build, with separate components handling feature extraction, acoustic modeling, pronunciation, and language modeling. Around 2018, systems began moving toward end-to-end neural networks that learned to map audio directly to text. But even this improvement left a gap: the model still couldn't interpret tone, emotion, or speaking speed without additional engineering.

Google's new approach addresses this by training models natively on multiple modalities at once. Instead of adding audio to a text-based system, the company builds models that learn relationships between audio, video, and text simultaneously during pretraining. A single training example might combine a text instruction with video and audio sequences, teaching the model to understand how these signals relate within one unified task.

How Are These New Models Actually Different?

Google has introduced two new live dialogue models designed to make voice interactions feel more natural and intelligent. Gemini 3.8 Live prioritizes speed and cost efficiency, combining conversational intelligence with fluid dialogue and visual grounding. Gemini 3.8 Live Extended Thinking handles high-complexity tasks with increased reasoning and multi-step problem-solving.

The performance improvements are substantial. Gemini 3.8 Live Extended Thinking captured the top overall spot on Artificial Analysis' Speech to Speech Quality Index with a score of 82.6, and leads in agentic task completion with 68.6% on one benchmark and 35.1% on another banking-focused benchmark. It also scored 97.7% on Big Bench Audio, a widely used reasoning benchmark, while maintaining competitive pricing compared to other frontier models.

What makes these models feel different in practice is their ability to handle real-world complexity. Gemini 3.8 Live processes visual inputs in near real-time, automatically detects and transitions between 97 supported languages mid-conversation, and executes tools and API calls in the background while continuing to chat. For tasks requiring deeper reasoning, 3.8 Live Extended Thinking reasons and speaks simultaneously, using natural verbal cues like "Let me check that..." to acknowledge requests while working through multi-step problems.

What Are the Key Capabilities These Models Deliver?

• Real-time visual grounding: Processes images and video feeds instantly to provide context-aware responses, enabling applications like live employee onboarding guidance and real-time visual problem-solving.
• Multilingual fluency: Automatically switches between 97 languages mid-conversation while preserving the original speaker's voice and understanding multiple speakers simultaneously, even in noisy environments.
• Background task execution: Runs tools, API calls, and complex workflows in the background without interrupting natural conversation flow, allowing the model to acknowledge requests and keep talking while tasks complete.
• Reasoning with narration: For complex tasks, the model thinks and speaks simultaneously, providing live progress updates that walk users through multi-step processes as they happen.
• Streaming translation quality: Delivers translation quality comparable to offline systems that have access to complete utterances, despite working in real-time with incomplete information.

The underlying technology relies on what Google calls "native multimodal pretraining." Rather than bolting audio onto a text-based language model, the company trains models from the ground up on interleaved examples combining text, audio, and video. This teaches the model to understand relationships among all three modalities within a single token embedding space, enabling it to move fluidly between understanding audio, generating speech, translating, and acting on instructions.

How Is Google Expanding Language Coverage to Underrepresented Languages?

Google's ambition extends far beyond the dominant languages where AI already performs well. The company has launched the 1,000 Languages Initiative, aiming to support the world's 1,000 most-spoken languages. This requires rethinking how AI gathers training data, since the web disproportionately represents a handful of dominant languages.

The solution involves grassroots partnerships with local communities. Google's Universal Speech Model was trained on 12 million hours of audio and uses cross-lingual transfer learning, a technique that enables models to apply patterns learned from data-rich languages to improve speech understanding in languages with far less training data.

Three major open-data initiatives exemplify this approach. WAXAL, which means "speaking" in Wolof, is a large-scale open speech dataset covering 27 Sub-Saharan African languages spoken by more than 100 million people across more than 26 countries, capturing tonal variation and conversational rhythms often missing from traditional datasets. Project Vaani, developed with the Indian Institute of Science and Bhashini, has collected more than 30,000 hours of speech across 109 languages from more than 155,000 speakers using a region-anchored approach. The Amplify Initiative brought together more than 1,600 local experts and 20 universities across four continents to contribute 15,000 multimodal data points capturing local nuance.

Google is also building Language Explorer, an interactive tool that visualizes LinguaMeta, the world's largest open-source language data repository. This tool continuously maps more than 7,000 spoken, written, and signed languages, recognized by Fast Company for design innovation.

What About People Without Reliable Internet or Modern Phones?

For more than 3 billion people, reliable internet access remains out of reach. Google developed TranslateGemma, a family of lightweight open translation models trained across 55 languages that run efficiently on-device, eliminating the need for cloud connectivity or internet access.

For the hundreds of millions still using feature phones in low-resource regions, Google is supporting organizations like Viamo to power "Ask Viamo Anything" (AVA), a voice AI assistant that brings Gemini's capabilities to standard feature phones. Viamo successfully piloted AVA in Rwanda and the service has already used Gemini to answer more than 2 million questions.

Google is also designing for accessibility from the ground up. Sign Language-to-Text (SL2T), trained across 50 plus sign languages, powers sign-to-text dictation in Gboard and Live Transcribe on Pixel 11, starting with American Sign Language to English. This represents an important first step toward making AI tools accessible to the 70 million people worldwide who rely on sign language to communicate.

What Trade-offs Are Engineers Still Navigating?

Building voice agents that excel across multiple dimensions presents genuine tensions. Valeria Wu Fon, who leads product for Gemini's speech-to-speech model, identified three competing priorities: conversational models should respond with low latency and feel snappy and natural; intelligent models should complete tasks, follow instructions, and reason well enough to deliver useful outcomes; and multimodal models should accept more than speech, including video, screen sharing, and PDFs.

"Increasing the thinking budget lets the model spend longer reasoning before answering or calling a tool, and this improves intelligence evaluations. But that work delays the first audio response and can make the conversation feel less natural," noted Valeria Wu Fon, leading product for Gemini's speech-to-speech model.

Valeria Wu Fon, Product Lead, Gemini Audio Team at Google DeepMind

Time to first audio is therefore a product constraint alongside answer quality. The team's aim is to combine all three priorities without large sacrifices, though the research acknowledges this balance remains an active challenge.

Language coverage cuts across all three priorities. The majority of Gemini users are non-English speakers, so the team focuses on making capabilities work beyond US English and across the languages customers actually use. A model's conversational or reasoning quality cannot be judged solely through its English behavior.

How to Start Using These New Voice Models

• For developers: Access Gemini 3.8 Live and 3.8 Live Extended Thinking through the Gemini API and Google AI Studio to build and deploy high-performance voice-driven interfaces using developer platforms like Agora, Fishjam, LangChain, LiveKit, Pipecat, Vercel, and Vision Agents.
• For enterprises: Gemini 3.8 Live is available now, while 3.8 Live Extended Thinking is in private preview in Gemini Enterprise and coming soon to Gemini Enterprise for Customer Experience and Google Workspace business customers.
• For everyone: Gemini 3.8 Live is available in Search Live, while 3.8 Live Extended Thinking is available in Gemini Live and for Google AI Pro and Ultra subscribers in Workspace in Docs, and all Google AI subscribers in Gmail and Keep.
• For safety: All audio generated by Google's AI products is watermarked with SynthID, an imperceptible watermark woven directly into audio output to ensure AI-generated content remains detectable and help prevent misinformation.

The shift from text-based translation to native audio intelligence represents a fundamental change in how AI understands human communication. By training models to process audio, video, and text together from the start, Google is building systems that capture not just what people say, but how they say it, why they say it, and what they mean. For billions of people speaking languages that have historically been left out of technology, this approach could finally make AI feel like it's actually listening.