A new framework tackles a fundamental problem in multimodal AI: knowing when to trust audio, when to trust video, and when to ignore one entirely. Researchers at the University of Cambridge and Queen's University Belfast developed a system that detects whether a person is actually visible and audible in video footage before attempting to match them, dramatically improving retrieval accuracy in real-world broadcast archives.

Why Does It Matter When One Modality Is Missing?

Standard AI systems trained on curated datasets assume both voice and face data are always present and reliable. But real broadcast archives tell a different story. A journalist might narrate footage without appearing on screen. An interview subject might be visible but silent. A crowd scene might show faces without clear audio attribution. When AI systems blindly fuse audio and video scores in these scenarios, they inject noise that actually makes results worse than using a single modality alone.

The BBC Rewind corpus, a collection of over 12,000 broadcast videos spanning 1948 to 1979, revealed the scope of this problem. Researchers identified three distinct presence types across the archive: audio-visual presence (both seen and heard), audio-only presence (heard but not visible), and visual-only presence (visible but not speaking). When a system tried to match a query person using both modalities indiscriminately, it performed worse than using just the best single modality.

How Does the New System Detect Active Modalities?

The breakthrough lies in cross-modal consistency checking. The system compares how well audio and video agree on which archive files match a query. When both modalities are genuinely active, they should rank similar files highly. When one modality is absent, that modality's scores become noise, and agreement breaks down. By analyzing this agreement pattern, classifiers can determine whether both modalities are truly present.

The approach uses speaker embeddings extracted from audio via a pre-trained model trained on over 2,000 hours of speech data, and face embeddings extracted from video frames using advanced face detection and clustering. Rather than inventing new embedding techniques, the system focuses on the fusion strategy itself, adapting how much weight to give each modality based on whether it's actually informative.

What Results Did the Researchers Achieve?

On the BBC Rewind corpus, the adaptive system achieved 94.2% precision at rank 1, meaning it correctly identified the target person as the top result in over 94% of queries. This outperformed speaker-only retrieval (82.9% accuracy), face-only retrieval (93.4% accuracy), and fixed-weight fusion that treats all queries the same way (90.0% accuracy). The system's modality detection itself reached 89% accuracy, successfully identifying whether audio, video, or both were active in the query.

Perhaps most tellingly, the adaptive approach recovered 64% of the gap between fixed fusion and an oracle system with perfect knowledge of which modalities were present. This suggests the method is approaching the theoretical ceiling of what's possible without ground-truth labels.

How to Implement Adaptive Modality Detection in Your Workflow

• Analyze Cross-Modal Agreement: Before fusing audio and video scores, compute how consistently both modalities rank the same results. High agreement indicates both modalities are informative; low agreement signals a missing or unreliable modality.
• Train Presence Classifiers: Use within-modal and cross-modal cosine similarity scores as features to train classifiers that detect whether audio, video, or both are active for each query, rather than assuming all queries have both modalities.
• Adapt Fusion Weights Dynamically: Instead of using fixed weights to combine audio and video scores, adjust the weights based on detected modality presence. Downweight or exclude absent modalities entirely to avoid degrading precision.
• Validate on Real-World Data: Test your system on actual broadcast or archival footage where modality presence varies naturally, not just on curated benchmark datasets where both modalities are guaranteed to be present.

The research addresses a gap that prior work largely overlooked. Earlier studies on audio-visual fusion, including work on multimodal video search frameworks, had assumed both modalities were simultaneously informative. Some research demonstrated that multimodal fusion outperforms single-modal retrieval on the BBC Rewind corpus, but none had tackled the scenario where one modality might be entirely absent or uninformative.

This work has direct implications for journalism, forensics, and media indexing applications where locating a specific individual across large video archives is critical. Broadcast archives, legal evidence repositories, and news libraries all contain footage where people appear in varied contexts: sometimes speaking on camera, sometimes narrating off-screen, sometimes appearing silently in background footage. The adaptive approach makes these real-world retrieval tasks significantly more reliable.

The framework builds on the Multimodal Video Search by Examples (MVSE) system, an EPSRC-funded project for content-based retrieval in the BBC Rewind archive. By combining speaker embeddings extracted via speaker diarization and face embeddings extracted via frame-level detection and clustering, the system creates a foundation for query-adaptive fusion that learns when to trust each modality.