A new AI model called Gander can listen, speak, and reason all at the same time, mimicking how humans actually have conversations instead of waiting for one person to finish before the other responds. Developed by researchers from Tencent, Zhejiang University, Shanghai Jiao Tong University, The Chinese University of Hong Kong, and Nanyang Technological University, Gander represents a significant shift in how AI systems interact with people.

Unlike most AI assistants today, which operate in a turn-based format where you ask a question and wait for a complete response, Gander processes streaming inputs from video, speech, and text all at once. This means the model can be interrupted mid-sentence, can ask follow-up questions on its own, and can provide feedback while you're still talking, creating a genuinely conversational experience.

How Does Gander's Architecture Enable Real-Time Interaction?

The key to Gander's natural interaction lies in its two-part brain design. The system uses what researchers call a "Cerebellum-Brain collaborative framework." Think of it this way: the Cerebellum handles the quick, responsive conversational tasks, while the Brain tackles complex reasoning and planning in the background. The two parts communicate continuously, allowing the model to respond instantly while still thinking through harder problems.

To make this work technically, Gander flattens all incoming and outgoing information into what researchers call a "chunk-level token stream." In simpler terms, the model treats every piece of information, whether it's a sound, a word, or a video frame, as a sequence of tiny units it can process in order. This unified approach allows the system to maintain low latency, meaning it responds quickly without lag.

What Real-World Scenarios Can Gander Handle?

The researchers tested Gander across four key dimensions to measure its effectiveness. The model was evaluated on conversational ability, omni understanding (how well it processes multiple types of input), interactive capability, and agentic intelligence (its ability to plan and execute tasks). The results showed that Gander maintains natural, expressive spoken dialogue while handling complex multimodal scenarios.

Importantly, Gander proved robust in challenging real-world conditions that would trip up many current systems:

• Background Noise: The model maintains performance even when there is significant ambient noise in the environment, a common problem in everyday settings.
• Multi-Party Interactions: Gander can handle conversations involving multiple speakers, understanding who is talking and responding appropriately.
• Backchannel Communication: The model recognizes and responds to brief acknowledgments like "mm-hmm" or "I see" that people naturally use while someone else is speaking, without interrupting the flow.

How Was Gander Trained to Understand Natural Conversation?

Creating a model that truly understands human conversation required building a massive training dataset from scratch. The researchers constructed what they call a "large-scale training corpus" organized into four families of data, each teaching the model different aspects of interaction.

The speech interaction data alone makes up about 37 percent of the training material. This includes everything from general dialogue and spoken instructions to full-duplex interaction, where both people can speak simultaneously. To create realistic full-duplex examples, researchers used a four-stage pipeline: they collected dialogue scenarios, annotated where interruptions and acknowledgments naturally occur, rendered the audio using voice synthesis, and then performed quality control to ensure the examples were realistic.

The team created over 11,000 dialogue scenarios spanning 45 different everyday and task-oriented situations, from education and healthcare to travel, finance, and customer support. Each dialogue was designed to include natural interaction patterns, such as when a user might interrupt to clarify something or when the assistant might ask a follow-up question.

What Makes Gander Different From Current AI Assistants?

Current AI systems, including most large language models (LLMs), which are AI systems trained on vast amounts of text, rely on external tools to manage when to listen and when to speak. These external tools often fail in noisy environments or when multiple people are talking. Gander, by contrast, has interactivity built directly into its core design, making it inherently better at handling the messy reality of human communication.

The researchers emphasize that this shift from turn-based to continuous interaction represents a fundamental change in how AI should engage with people. Rather than treating conversation as a series of isolated exchanges, Gander treats it as a continuous stream where interruptions, overlaps, and spontaneous feedback are expected and handled naturally.

The team has released Gander along with its models, code, and training data to the research community, making it possible for other researchers and developers to build on this work and explore new applications for real-time multimodal AI interaction.