A tiny tweak to how AI models weight their learning signals can slash the computing power needed for reasoning tasks by up to 19 times. Researchers at Carnegie Mellon University discovered that normalizing the learning signal by successful attempts rather than total attempts causes AI models to learn math and coding problems far more efficiently, a finding that could reshape how companies train reasoning-focused AI systems.

Why Standard AI Training Misses the Bigger Picture?

Training AI models to solve math and coding problems is deceptively difficult. Most reinforcement learning approaches reward models for getting answers right on the first try, but this optimization strategy only captures part of the story. Fahim Tajwar, a PhD student in Carnegie Mellon's Machine Learning Department, and his collaborators identified a fundamental mismatch between what standard training optimizes for and what actually matters in practice.

The team's framework, called Maximum Likelihood Reinforcement Learning (MaxRL), shows that the practical change amounts to a single line in the advantage calculation. The mathematical basis comes from a Maclaurin expansion of the log-likelihood objective, which reveals that standard reinforcement learning only optimizes for pass@1 performance. In contrast, MaxRL optimizes a harmonic mixture of successively higher pass@k gradients, meaning harder problems where correct solutions are rare receive stronger learning signals rather than being effectively ignored.

How Does MaxRL Improve AI Reasoning Performance?

The researchers tested MaxRL across four compute tiers, each designed to validate the approach progressively while remaining accessible to research groups with different computational resources. The experiments ran on ACCESS GPU resources including NCSA Delta and NCSA DeltaAI, with tasks ranging from simple image classification to large-scale language model training.

• ImageNet Classification: Runnable on a single GPU, with a full experiment taking 10 to 15 hours depending on GPU type.
• Maze Navigation: Requires autoregressive generation from a transformer, demanding one full node of 4 graphics processing units (GPUs) for 48 to 72 hours.
• Math Reasoning (GSM8K): Requires 2 nodes of 4 GPUs or 1 node of 8 GPUs for up to 96 hours of training.
• Large-Scale Language Model Training: Requires 2 to 4 nodes of 4 GPUs for approximately 1 to 1.5 weeks per full training run.

The results were striking. MaxRL consistently matched or exceeded GRPO (a competing approach) on pass@1 accuracy while dramatically improving pass@k performance. This means trained models find correct answers far more reliably when given multiple attempts. Achieving the same coverage as MaxRL required GRPO to use 7.9 to 19.2 times more samples at test time, a massive efficiency gap.

MaxRL also showed less overfitting to the training distribution and maintained more solvable problems throughout training, enabling continued learning in later epochs. This suggests the approach doesn't just squeeze more performance out of existing compute; it fundamentally changes how models learn to reason.

What Practical Challenges Did the Team Face?

Scaling experiments across two clusters brought real-world complications that reveal how test-time compute research actually works in practice. The most significant technical issue was package compatibility. While building Flash-Attention and vLLM is straightforward on Delta, DeltaAI's ARM-based CPUs often lack pre-built package wheels, forcing the team to compile from source and later use pre-compiled aarch64 wheels available as open-source modules.

Multi-node communication and checkpoint management proved non-trivial, since MaxRL's longer training runs often exceed a single job allocation. The team used Globus to handle data and checkpoint transfers between Delta and DeltaAI reliably. They also discovered an unexpected inefficiency through monitoring tools. XDMoD revealed that vLLM, which is optimized for large models with 1 billion or more parameters, introduced significant inefficiencies when applied to small 3-million-parameter Maze models. This insight led the team to build a custom inference system for the Maze experiments, recovering wasted compute hours that would otherwise have gone undetected.

"The team's recommended approach: find a smaller, representative setting that runs on a single node (8 GPUs or 4 GPUs at most), validate the approach there, then scale up incrementally, the same ladder the paper itself follows," noted researchers at Carnegie Mellon.

Fahim Tajwar, PhD Student, Machine Learning Department, Carnegie Mellon University

How Can Researchers Adopt MaxRL for Their Own Work?

For researchers looking to implement MaxRL, the team recommends starting with an Explore or Discover tier allocation on ACCESS resources. The key is finding a smaller, representative setting that runs on a single node with 8 GPUs or 4 GPUs at most, validating the approach there, then scaling up incrementally. This ladder approach mirrors the structure of the paper itself, making it easier to debug issues and understand where improvements come from.

The team's solutions, including proper use of the Ray Python package for resource management and automatic checkpoint resumption, are open-sourced in the MaxRL code repository. This means other researchers can build on their work without reinventing the infrastructure challenges they solved.

The broader implication is significant for the field. As AI labs increasingly focus on test-time compute, inference scaling, and reasoning at test time, understanding how to train models more efficiently becomes critical. MaxRL demonstrates that sometimes the most impactful improvements come not from throwing more compute at a problem, but from rethinking the fundamental objective that guides learning. A single line of code, grounded in solid mathematical reasoning, can unlock efficiency gains that would otherwise require nearly 20 times more computational resources.