Audio AI models excel at transcribing spoken words but frequently misinterpret emotional tone, pitch, and other nonverbal cues that convey true meaning. A new study from USC researchers found that even the most advanced audio large language models (LLMs), which are AI systems trained on vast amounts of text and audio data, treat spoken language as secondary to written text, causing them to miss critical paralinguistic information that shapes human communication.

The research, led by USC professor Mohammad Soleymani, examined why conversational AI tools like ChatGPT and Gemini struggle to understand how something is said, not just what is said. The team's paper, "Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox," was accepted to the International Conference on Machine Learning (ICML) 2026.

Why Do Audio Models Fail at Understanding Emotion and Tone?

Audio LLMs are multimodal AI models designed to process both text and audio inputs, including speech and environmental sounds. However, these systems suffer from a fundamental design flaw: they prioritize language over acoustic information. When a speaker's tone contradicts their words, the models typically side with the text.

For example, if someone says "I am sad" in a clearly happy tone, an audio LLM will incorrectly conclude the person is sad, ignoring the emotional cues embedded in their voice. This limitation poses real risks in sensitive applications such as health assessments or mental health support, where misreading a user's emotional state could lead to harmful responses.

"Even today's most advanced audio large language models struggle to interpret information beyond the spoken words, often taking language too literally and missing the nonverbal cues that have an equally important role in human communication," explained Mohammad Soleymani, a research associate professor at USC Viterbi School of Engineering and the USC Mark and Mary Stevens School of Computing and AI's Thomas Lord Department of Computer Science.

Mohammad Soleymani, Research Associate Professor at USC Viterbi School of Engineering

To understand why this happens, Soleymani's team, which included PhD students Ashutosh Chaubey and Jiacheng Pang, examined the internal workings of audio LLMs. They discovered a phenomenon called "representation degradation," where acoustic details like pitch and tone gradually disappear as audio data moves through the model's layers toward its language-processing components. By the time information reaches the final decision-making layer, much of the acoustic information has been stripped away.

The researchers also identified what they call a "utilization gap." Even when correct acoustic information exists within the model's internal representations, the model's decision-making process is so heavily biased toward language that it simply ignores the available audio evidence.

How Did Researchers Test This Problem?

To measure this blind spot, Soleymani's team created a benchmark called VoxParadox, which evaluates models across 10 different paralinguistic tasks. These tasks span biometric challenges, such as estimating a speaker's age or gender and counting how many people are speaking, as well as prosodic and acoustic tasks, including identifying emotion, intonation, pitch, and volume.

For each task, researchers presented audio clips where spoken words intentionally contradicted the accompanying audio content and asked the models to determine whether the uttered words were true or false. If the model answered incorrectly, it suggested the model was prioritizing text over audio. If it answered correctly, it demonstrated that the model had incorporated nonverbal audio information into its reasoning.

The team evaluated NVIDIA's Audio Flamingo 3 and Alibaba's Qwen2-Audio models and found that both struggled significantly to accurately interpret paralinguistic information. The results revealed a critical vulnerability in how these widely used audio AI systems process human speech.

How to Improve Audio AI's Ability to Understand Tone and Emotion

Soleymani's team proposed a two-part solution designed to help audio LLMs both retain and value what they hear:

• Prompt-Conditioned Layer Mixer (PCLM): This module helps the model determine exactly which part of the audio it should "listen" to based on the question being asked. Instead of relying only on the encoder's final layer, PCLM draws information from multiple layers simultaneously, allowing the model to access both early layers that capture basic acoustic information like pitch and volume, and deeper layers that specialize in understanding vocabulary and language.
• Direct Preference Optimization (DPO): This post-training method aligns the model's decision-making process with the available acoustic evidence. During DPO training, the model is shown pairs of responses and learns which answer should be preferred because it better matches the audio, encouraging it to rely less on textual shortcuts and more on what it actually hears.
• Adaptive Task Recognition: PCLM examines the user's text prompt to determine the optimal combination of audio information needed for a particular task. If a user asks "Is this speaker angry?" the system recognizes that the task depends more heavily on emotional cues than on spoken words and adjusts its processing accordingly.

"Just like a hearing aid, PCLM boosts the volume of the audio cues that are most relevant to your question," said Soleymani.

Mohammad Soleymani, Research Associate Professor at USC Viterbi School of Engineering

Together, PCLM and DPO produced dramatic improvements. After incorporating both methods, the models' ability to interpret paralinguistic information improved significantly, with accuracy increasing from 17% to 65%. This represents a nearly four-fold improvement in the models' ability to understand emotional tone, pitch, and other nonverbal communication cues.

The study is the first to both identify this blind spot in audio LLMs and propose practical solutions to address it. By opening the door for other researchers to recognize and tackle this problem, Soleymani's work could reshape how conversational AI systems are designed and trained, making them more attuned to the full spectrum of human communication.