A new benchmark called SuperARC reveals that leading large language models (LLMs) are not progressing toward artificial general intelligence (AGI) as their developers claim. Published in Nature, the study introduces a test based on fundamental mathematical principles rather than human-centric questions, exposing a critical gap between what AI companies market and what their models actually achieve.

What Makes SuperARC Different From Every Other AI Test?

Most AI benchmarks today rely on human-written questions with expected answers, or they measure how well models match patterns in training data. SuperARC takes a radically different approach. Instead of asking "What is the capital of France?" the test grounds itself in Algorithmic Information Theory (AIT), a branch of mathematics that defines randomness, prediction, and optimal reasoning using universal principles discovered by researchers like Gregory Chaitin and Ray Solomonoff.

Think of it this way: traditional benchmarks ask if an AI can memorize facts or recognize patterns. SuperARC asks whether an AI can truly compress information and extract meaning from it, which researchers argue is the actual foundation of intelligence itself. The test evaluates models on their ability to identify recursive patterns, make predictions from those patterns, and generate concise explanations for complex phenomena.

Why Are Today's Best AI Models Failing?

The results are sobering. When researchers applied SuperARC to frontier models, the leading LLMs outperformed most competitors in multiple tasks, but they did not consistently improve across newer model versions. In fact, some of the latest versions actually regressed, performing worse than earlier iterations. This contradicts the narrative that each new model release represents progress toward AGI.

The core problem, according to the research, is that LLMs excel at statistical pattern matching but struggle with true compression-based reasoning. The study found that predictive power through formal theories is directly proportional to compression over the algorithmic space, not the statistical space. In simpler terms, models that can truly understand and compress information algorithmically outperform those that merely recognize statistical patterns.

Interestingly, the researchers demonstrated that a hybrid neuro-symbolic approach, combining neural networks with symbolic reasoning, outperformed specialized prediction models in relevant examples related to compression and sequence prediction. This suggests that the path forward may not be scaling up LLMs alone, but rather integrating them with symbolic methods that many AI developers are already adopting, often without fully acknowledging it.

How to Evaluate AI Models Beyond Marketing Claims

• Benchmark Orthogonality: Look for tests that are fundamentally different from existing benchmarks rather than variations on the same theme. SuperARC is orthogonal to current tests because it measures compression-based reasoning rather than pattern matching or factual recall.
• Mathematical Grounding: Prefer benchmarks rooted in formal mathematical principles like Algorithmic Information Theory rather than human-centric metrics that reflect biological quirks or shared human history.
• Version Consistency: Examine whether newer model versions actually improve on rigorous tests or merely claim superiority through marketing. Regression in newer versions is a red flag that progress may be illusory.
• Hybrid Approach Assessment: Evaluate whether models incorporate symbolic reasoning alongside neural components, as this combination appears more aligned with true reasoning than pure statistical approaches.

The Bigger Picture: AI Test Saturation and the Need for New Metrics

The research highlights a critical problem in AI development: test saturation. As all major AI companies claim to be "best in class" on available benchmarks, those benchmarks lose their ability to differentiate genuine progress from marketing hype. The study argues that new tests orthogonal to current ones must continue to be developed to keep challenging increasingly sophisticated models.

This mirrors a historical pattern in human intelligence testing. For decades, IQ tests were considered the gold standard for measuring intelligence, yet they reflected human-centric biases and biological peculiarities rather than objective measures of reasoning. The same risk exists with AI benchmarks today. By relying too heavily on tests that measure human-like abilities, researchers may be overlooking what true machine intelligence actually requires.

The researchers acknowledge that grounding intelligence metrics in fundamental computation and mathematics is not without philosophical challenges. By reducing intelligence to observable outputs, there is a risk of overlooking internal representation, semantic understanding, or other dimensions that may matter. However, they argue that this mathematical foundation offers a more objective framework than human-centric alternatives.

What This Means for AI Development Going Forward

The SuperARC findings suggest that the AI industry's current trajectory may be misleading. Companies racing to build larger models and claim AGI capabilities may be optimizing for the wrong metrics. If true reasoning requires compression-based understanding rather than statistical pattern matching, then scaling up existing LLM architectures alone will not bridge the gap to genuine artificial general intelligence.

The study concludes that further progress in AI models can only be achieved in combination with symbolic approaches that integrate formal reasoning with neural learning. This represents a significant departure from the pure deep learning paradigm that has dominated the field for the past decade. It suggests that the next generation of AI breakthroughs may come not from bigger models, but from smarter architectures that blend neural and symbolic methods.

As the AI field enters this test saturation phase, SuperARC offers a template for how future benchmarks should be designed: grounded in universal mathematical principles, orthogonal to existing tests, and capable of distinguishing genuine progress from incremental improvements or marketing claims. For researchers, developers, and anyone evaluating AI capabilities, this work provides a crucial reality check on what current models can and cannot do.