Leading AI language models can identify the sentiment of Chinese idioms with over 95% accuracy, but struggle dramatically when asked to use them appropriately in real-world contexts, achieving only around 40% accuracy on open-ended tasks. Researchers have introduced Chengyu-Bench, a comprehensive evaluation tool that exposes a critical blind spot in how today's most advanced AI systems understand one of the world's most culturally rich languages.

Chinese idioms, known as Chengyu, are four-character expressions rooted in centuries of history, folklore, and classical literature. Unlike English idioms, their meanings cannot be deduced by simply translating the individual characters. For example, the idiom "亡羊补牢" literally translates to "mend the fence after sheep are lost," but actually means "it is never too late to try." This metaphorical nature, combined with their fixed structure and deep cultural context, makes them uniquely challenging for artificial intelligence systems trained primarily on modern text data.

What Makes Chinese Idioms So Hard for AI to Master?

Researchers identified several fundamental challenges that prevent even state-of-the-art language models from truly grasping Chinese idioms. These obstacles reveal why raw pattern recognition alone is insufficient for language understanding.

• Non-compositional meaning: Idioms cannot be understood by breaking them into individual characters or words; their meanings emerge from cultural and historical context that models must learn separately.
• Fixed structure constraints: Chinese idioms follow rigid four-character formats that cannot be decomposed or rearranged, limiting how models can process and manipulate them.
• Cultural and historical grounding: Many idioms derive from ancient anecdotes and classical texts, requiring deep knowledge of Chinese tradition that modern training data may not adequately represent.
• Context-dependent usage: The same idiom can carry different connotations depending on context, and closely related idioms often have subtle differences that models frequently confuse.

How Does Chengyu-Bench Evaluate AI Performance?

The new benchmark, developed by researchers studying language understanding, includes over 3,000 human-verified examples covering 1,765 common Chinese idioms sourced from diverse, naturally occurring texts. Rather than relying on simple multiple-choice tests, the benchmark features three progressively challenging tasks that reflect real-world usage.

The first task, Evaluative Connotation, asks models to classify idioms as positive or negative. This is where AI systems perform best, achieving over 95% accuracy. The second task, Appropriateness, requires models to detect when an idiom is used incorrectly in context, a skill that drops performance to approximately 85% accuracy. The third task, Open Cloze, presents the most difficulty: models must generate the appropriate idiom to fill a blank in a longer passage without multiple-choice options, where they achieve only around 40% top-1 accuracy.

Error analysis reveals that most mistakes arise from fundamental misunderstandings of idiom meanings rather than minor contextual confusion. This suggests that current AI systems lack the deep cultural grounding necessary to use idioms naturally, even when they can recognize them in isolation.

Steps to Improve AI's Understanding of Culturally Rich Language

The research points toward several approaches that could help AI systems better grasp the nuances of Chinese idioms and similar culturally embedded language features:

• Expand training data diversity: Models need exposure to idioms in varied contexts from classical literature, modern media, and everyday communication to develop a richer understanding of how they are actually used.
• Incorporate cultural knowledge explicitly: Rather than relying solely on pattern matching, AI systems could benefit from structured knowledge about the historical origins and cultural significance of idioms.
• Design task-specific evaluation benchmarks: Comprehensive benchmarks like Chengyu-Bench that test recognition, appropriateness, and generation separately reveal gaps that simpler tests miss, guiding future model improvements.

Why This Matters Beyond Chinese Language

The findings from Chengyu-Bench highlight a broader challenge in artificial intelligence: the difference between recognizing patterns and truly understanding language. While AI systems have become remarkably good at statistical pattern matching, they often lack the cultural and contextual grounding that humans develop naturally through lived experience.

This gap has practical implications for any AI application that needs to communicate naturally in languages rich with idioms, metaphors, and cultural references. Machine translation systems, chatbots, and content generation tools all struggle when they encounter language that cannot be understood through literal word-by-word analysis. As AI systems are deployed globally, this limitation becomes increasingly important to address.

The researchers have made both the Chengyu-Bench benchmark and its source code publicly available, inviting the broader AI research community to use these tools to develop better models. This collaborative approach could accelerate progress toward AI systems that truly understand not just the words of a language, but the culture and history embedded within it.