Multimodal AI models that can see and hear are surprisingly bad at counting things in long videos, new research shows. When asked to count specific objects or events across 30-minute-plus videos, the best AI systems achieve only 23.74% accuracy, compared with human performance of 82.97%. But the real discovery isn't just that AI fails at counting; it's why.

Researchers from a new study built EC-Bench, an evaluation suite designed to diagnose exactly where multimodal large language models (MLLMs) break down when processing long-form video. MLLMs are AI systems that combine language understanding with the ability to process images and video. The team evaluated 22 different open-source and proprietary models using videos longer than 30 minutes paired with transcripts, asking them to count specific instances and explain their reasoning.

What's Actually Going Wrong When AI Tries to Count?

The research reveals that counting failures aren't simple arithmetic mistakes. Instead, the problem breaks down into three interconnected challenges that AI systems struggle with simultaneously. When an AI model tries to count objects or events in a long video, it must first identify every relevant instance, then pinpoint exactly when each one appears in the video timeline, and finally aggregate all those findings into an accurate total.

The study found that when AI systems miss instances or double-count the same event, it's almost always because they failed at one of these earlier steps. The researchers discovered that enumeration accuracy (correctly identifying all relevant instances) is strongly linked to final counting accuracy. When a model struggles to find all the instances it's supposed to count, the counting error naturally follows.

Long videos create a unique problem that short-video AI systems don't face. Relevant evidence can be sparse, visually diverse, and separated by tens of minutes of unrelated content. A model must search through an entire untrimmed video, apply temporal or semantic constraints, avoid counting the same instance twice, and maintain consistency across the entire timeline. This is fundamentally different from simply predicting a number.

How to Understand What AI Models Are Actually Doing Wrong?

• Enumeration Failure: The model fails to identify all query-relevant instances in the video, achieving only 29.98% F1 score (a measure of accuracy) compared with human performance of 78.57%. This means AI systems are missing instances or including irrelevant ones.
• Temporal Grounding Weakness: Even when a model identifies an instance, it struggles to pinpoint exactly when that instance occurs in the video timeline, which directly correlates with higher counting errors.
• Deduplication Problems: When the same event or object appears multiple times or spans across video boundaries, AI systems often count it more than once or fail to recognize it's the same instance.
• Evidence Distribution Challenge: Counting accuracy drops significantly when supporting evidence is spread throughout the video rather than clustered together, forcing the model to maintain context across longer temporal gaps.

The EC-Bench dataset itself represents a significant contribution to AI research. It contains 152 untrimmed videos, each longer than 30 minutes, paired with 1,699 open-ended counting queries across six different reasoning categories. Crucially, each query includes human-verified answers and temporal evidence spans that show exactly where in the video the correct answer comes from.

This level of annotation allows researchers to see not just whether a model got the final answer right, but which specific steps in the reasoning process failed. Previous video benchmarks typically only evaluated whether a model produced the correct final number, leaving a blind spot about what the model actually counted and where it found that information.

Why Does This Matter for Real-World AI Applications?

The implications extend beyond academic benchmarking. Long-form video understanding is increasingly important for practical applications like surveillance analysis, medical imaging review, sports analytics, and content moderation. In these real-world scenarios, you don't just need a correct final count; you need to understand what the AI actually counted and where it found that evidence, especially if the decision has consequences.

The research reframes long-video counting as a problem of evidence retrieval, temporal grounding, deduplication, and aggregation rather than simple numerical prediction. This distinction matters because it suggests that improving AI performance on these tasks requires different approaches than simply training models to predict numbers more accurately. Instead, developers need to focus on helping models systematically search videos, track evidence across time, and maintain consistent reasoning across extended temporal contexts.

The gap between current AI performance and human performance remains substantial. The best-performing models reach only about 30% accuracy on enumeration and 24% on final counting, while humans consistently achieve around 79% and 83% respectively. This suggests that current approaches to processing long-form video content have fundamental limitations that won't be solved by incremental improvements alone.