AI systems that analyze emotions from videos and audio are making fundamental mistakes, either fabricating evidence that doesn't exist or justifying feelings with completely irrelevant details. A new study from USC researchers has identified exactly where and why these systems fail, and proposes a solution to ground AI's emotional reasoning in actual evidence rather than logical shortcuts.

Why Do AI Models Get Emotions So Wrong?

Multimodal large language models (MLLMs) are advanced AI systems that can process multiple types of data simultaneously, including images, audio, video, and text. These models are increasingly used in customer service, mental health applications, and other scenarios where understanding human emotion matters. However, researchers found that today's AI systems are surprisingly bad at this task.

Mohammad Soleymani, a research associate professor at USC Viterbi School of Engineering, led a team that investigated why AI models fail to connect audiovisual cues with the correct emotions. The research, which began in spring 2026, resulted in a paper titled "AVERE: Improving Audiovisual Emotion Reasoning With Preference Optimization," accepted to the International Conference on Learning Representations (ICLR) 2026.

To understand the problem, the team created a benchmark called EmoReAlM consisting of 4,000 multiple-choice questions. AI models were shown video clips along with separate audio and visual captions, then asked to identify emotions and explain their reasoning. The researchers identified two distinct categories of failure.

What Are the Two Types of AI Emotion Errors?

The first failure mode involves reasoning errors. In these cases, the AI correctly identifies an emotion but justifies it using audiovisual cues that are completely unrelated to that feeling. For example, models might claim "This man is scared because he's bald" or "The girl is happy because her sweater is green." The cues actually exist in the video, but they have nothing to do with the emotion.

The second failure mode is perception errors, which are even more problematic. Here, the AI fabricates evidence that doesn't exist in the video at all. Models might claim a person is sad because crying sounds were detected, when the audio is actually completely silent. The researchers found that models were taking cognitive shortcuts, hyper-focusing on text prompts mentioning specific emotions like "sadness" and then generating a chain of reasoning that satisfied their own initial prediction rather than looking for real evidence.

• Reasoning Errors: AI identifies the correct emotion but justifies it with irrelevant audiovisual cues that are actually present in the video but unrelated to the feeling being expressed.
• Perception Errors: AI completely fabricates evidence that doesn't exist, hallucinating sounds or visual details to support its predetermined conclusion about an emotion.
• Language Bias: MLLMs treat text as more important than audio and visual information, causing them to prioritize what a caption says over what actually appears on screen.
• Ambiguous Emotions: Models performed poorly on emotions like "disgust," which researchers attributed to the inherent ambiguity of the emotion and insufficient training data for that specific category.

The team also discovered that models struggled when signals conflicted, such as when a caption mentioned "crying" but the person in the video was actually crying tears of joy.

"Our first step was to understand and quantify these challenges and pinpoint where and how exactly these multimodal models fail in the emotion reasoning process," explained Mohammad Soleymani.

Mohammad Soleymani, Research Associate Professor at USC Viterbi School of Engineering

How to Fix AI's Emotional Reasoning Problems

To address these failures, Soleymani's team developed a two-part solution designed to ground AI's reasoning in actual evidence rather than language shortcuts.

• Audiovisual Emotion Direct Preference Optimization (AVEm-DPO): This technique creates preference pairs showing the model a logically sound, grounded output versus a hallucinated or misattributed output, forcing the system to prioritize correct reasoning. After applying this method, attention maps showed that models shifted their focus to pay significantly more attention to audio and video information rather than just text prompts.
• Text Prior Debiasing (TPD): This approach specifically penalizes AI for providing answers that could be guessed from text alone, forcing the model to ground its reasoning in actual audiovisual evidence. Essentially, it tells the AI that responses generated purely from a text prompt are inferior to those using audio and visual data.
• Evidence-Based Validation: The framework ensures that AI can only justify emotional conclusions using cues that actually exist in the video or audio, eliminating the cognitive shortcuts that lead to hallucination and irrelevant reasoning.

The research was funded by the Army Research Office (ARO), the National Science Foundation (NSF), and the National Institute of Mental Health (NIMH). The team included Soleymani's PhD students Jiacheng Pang and Ashutosh Chaubey, as well as Maksim Siniukov.

These findings have significant implications for AI applications in customer service, mental health support, and any scenario where understanding human emotion is critical. As AI systems become more integrated into sensitive social interactions, ensuring they can reliably interpret emotions without hallucinating evidence becomes increasingly important. The USC team's work demonstrates that debugging AI's logic is possible, but requires deliberate optimization techniques that force models to prioritize real evidence over convenient shortcuts.