Audio AI models excel at transcribing what people say but fail to understand how they say it, missing tone, emotion, and other vocal cues that carry crucial meaning in human communication. A new study from USC researchers reveals that even the most advanced audio large language models (LLMs) treat spoken words as primary information while largely ignoring the acoustic signals that convey emotional context, sarcasm, and intent. The finding exposes a fundamental blind spot in multimodal AI systems that are increasingly embedded in conversational tools like ChatGPT and Gemini.

Why Are Audio Models Tone-Deaf?

Audio LLMs are designed to process both text and audio inputs, but they suffer from what researchers call a "strong language bias," treating text as the dominant signal and audio cues as secondary information. When speech is converted into numerical representations called vectors, the model's internal processing gradually strips away acoustic details like pitch, tone, and rhythm as the data moves through deeper layers toward language-processing components. By the time information reaches the model's final decision-making layer, much of the acoustic information has disappeared, a phenomenon researchers refer to as "representation degradation".

This creates a paradoxical situation: the correct acoustic information often still exists within the model's internal representations, but the model's decision-making process is so heavily biased toward language that it simply ignores the available audio evidence. The result is functionally "tone-deaf" AI. If a speaker sounds clearly happy but says the words "I am sad," the model will incorrectly conclude that the speaker is sad, missing the contradiction entirely.

USC professor Mohammad Soleymani led research that began last August to uncover why audio LLMs struggle with these listening cues. The team's paper, "Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox," was accepted to the International Conference on Machine Learning (ICML) 2026. The research team included Soleymani's PhD students Ashutosh Chaubey and Jiacheng Pang.

How Did Researchers Test This Problem?

To evaluate how well audio models understood paralinguistic information, Soleymani's team designed a series of "stress tests" and introduced a benchmark called VoxParadox, which evaluates models across 10 different paralinguistic tasks. These tasks span biometric and identity-related challenges, such as estimating a speaker's age or gender and counting how many people are speaking, as well as prosodic and acoustic tasks, including identifying emotion, intonation, pitch, and volume.

For each task, researchers presented audio clips where spoken words intentionally contradicted the accompanying audio content and asked models to determine whether the uttered words were true or false. If the model answered incorrectly, it suggested the model was prioritizing text over audio. If it answered correctly, it demonstrated that the model had incorporated nonverbal audio information into its reasoning. The team evaluated NVIDIA's Audio Flamingo 3 and Alibaba's Qwen2-Audio models and found that both struggled to accurately interpret paralinguistic information.

What Solutions Are Researchers Proposing?

Soleymani's team proposed a two-part solution designed to help audio LLMs both retain and value what they hear. The first component is a module called the Prompt-Conditioned Layer Mixer (PCLM), which helps the model determine exactly which part of the audio it should "listen" to based on the question being asked. As audio passes through an encoder, acoustic details such as pitch and tone are often suppressed in deeper layers as the model becomes increasingly focused on words. Instead of relying only on the encoder's final layer, PCLM draws information from multiple layers simultaneously before passing it to the language model.

"Just like a hearing aid, PCLM boosts the volume of the audio cues that are most relevant to your question," explained Mohammad Soleymani.

Mohammad Soleymani, Research Associate Professor at USC Viterbi School of Engineering

The team also applied a second technique called Direct Preference Optimization (DPO), a post-training method that aligns the model's decision-making process with the available acoustic evidence. During DPO training, the model is shown pairs of responses and learns which answer should be preferred because it better matches the audio, encouraging it to rely less on textual shortcuts and more on what it actually hears.

How Much Did These Solutions Improve Performance?

The results were dramatic. After incorporating both PCLM and DPO methods, the models' ability to interpret paralinguistic information improved significantly, with accuracy increasing from 17% to 65%. This represents a nearly fourfold improvement in the models' ability to understand how something is said, not just what is said. The study is the first to both identify this blind spot in audio LLMs and propose solutions to address it.

Why Does This Matter in Real-World Applications?

The implications extend far beyond academic research. In sensitive applications such as health assessments or human-AI interactions, failing to "hear" a user's true emotion or condition could result in incorrect or even harmful responses. A mental health chatbot that misses signs of distress in a user's voice, or a customer service agent that fails to recognize frustration in tone, could provide inappropriate guidance or escalate rather than resolve issues.

The challenge becomes more acute as audio AI systems become embedded in everyday tools. Users can now hold conversations with AI, ask questions aloud, send voice messages, and receive spoken responses in real time. Yet these systems remain fundamentally limited in their ability to interpret the full spectrum of human communication. Understanding paralinguistic cues is essential for AI to truly comprehend user intent and emotional state.

Steps to Improve Audio AI Interpretation

• Multi-Layer Information Access: Audio models should draw acoustic information from multiple processing layers rather than relying solely on final language-focused layers, allowing early acoustic-processing layers to contribute to decisions about tone and emotion.
• Task-Aware Attention Mechanisms: Models should examine user prompts to determine which audio signals are most relevant to the specific question being asked, similar to how PCLM performs "soft selection" of acoustic information.
• Post-Training Alignment: Models should undergo additional training that explicitly teaches them to value acoustic evidence alongside textual information, using preference optimization to reduce reliance on language shortcuts.

The research opens a new frontier in multimodal AI development. As audio LLMs become more prevalent in consumer applications, addressing the paralinguistic gap will be critical for building AI systems that truly understand human communication in all its complexity. The study demonstrates that the problem is not insurmountable; with the right architectural changes and training methods, audio models can learn to listen as well as they read.