Audio-visual artificial intelligence, which combines sound and sight into a single AI system, has emerged as a central frontier in AI research, with large foundation models now learning to perceive, generate, and reason across both modalities simultaneously. A new 56-page survey from researchers across leading institutions provides the first comprehensive framework for understanding this rapidly expanding field, consolidating fragmented research into a unified taxonomy that spans understanding, generation, and interactive tasks.

For years, AI systems treated audio and vision as separate problems. A model might excel at recognizing objects in images or transcribing speech, but integrating both capabilities into a coherent understanding of the world remained elusive. Recent breakthroughs from companies like Meta and Google, including systems such as Meta MovieGen and Google Veo-3, demonstrate that unified audio-vision architectures trained on massive multimodal datasets can now handle complex real-world scenarios where sound and sight are inseparable.

What Exactly Is Audio-Visual Intelligence in Foundation Models?

Audio-Visual Intelligence (AVI) represents a fundamental shift in how AI systems process information. Rather than treating audio and video as isolated streams, AVI models learn to understand how sound and sight interact in the real world. This matters because human perception is inherently multimodal; we don't just see a dog barking, we hear it too, and our brain integrates both signals to understand what's happening.

The survey identifies three broad categories of AVI tasks that foundation models now tackle. Understanding tasks include speech recognition, sound localization, and audio-visual event detection. Generation tasks span audio-driven video synthesis and video-to-audio conversion. Interaction tasks cover dialogue systems, embodied AI agents, and agentic interfaces that can both perceive and respond to multimodal inputs.

How Are Researchers Building These Unified Audio-Visual Systems?

• Modality Tokenization: Converting both audio and visual information into discrete tokens that a neural network can process, similar to how language models handle text, allowing a single architecture to handle multiple input types.
• Cross-Modal Fusion: Developing methods to combine audio and visual features at different levels of processing, enabling the model to understand how sound and sight relate to each other in time and space.
• Large-Scale Pretraining: Training foundation models on massive datasets containing paired audio and video, allowing them to learn general patterns about how the world looks and sounds before being fine-tuned for specific tasks.
• Autoregressive and Diffusion-Based Generation: Using two different approaches to generate new audio or video content, either by predicting one token at a time or by iteratively refining outputs from noise, depending on the task requirements.
• Instruction Alignment and Preference Optimization: Fine-tuning models to follow user instructions and match human preferences, making them more controllable and aligned with what people actually want the system to produce.

Why Should Anyone Care About Audio-Visual AI Right Now?

The practical implications are substantial. Consider accessibility applications: a system that truly understands both audio and visual information could provide richer descriptions for blind users or generate captions that capture not just what's said, but the emotional tone conveyed by a speaker's voice. In content creation, unified audio-visual models eliminate the need to use separate tools for audio and video, streamlining workflows for creators who currently juggle multiple AI systems.

The survey also highlights critical open challenges that researchers are actively working to solve. Synchronization between audio and video remains difficult; models must learn that a person's lips should move in sync with their speech, not randomly. Spatial reasoning requires understanding where sounds originate in a visual scene. Controllability means users need fine-grained control over what the system generates. Safety concerns include preventing models from generating harmful or misleading content, particularly when audio and video can be combined to create convincing deepfakes.

The research landscape has historically been fragmented, with inconsistent taxonomies and evaluation practices making it hard to compare progress across different labs and companies. This new survey establishes standardized benchmarks and evaluation metrics, curating representative datasets that researchers can use to measure progress fairly. By consolidating this rapidly expanding field into a coherent framework, the survey aims to serve as a foundational reference for future research on large-scale audio-visual intelligence.

The timing is significant. As foundation models grow more capable, the ability to process multiple modalities simultaneously is becoming table stakes rather than a novelty. The researchers behind this survey, including contributors from leading institutions, have documented how industrial and academic focus on unified audio-vision architectures continues to accelerate, driven by the recognition that real-world intelligence requires understanding the world through multiple senses at once.