A fundamental shift in how AI models are trained has emerged over the past year, moving beyond human feedback to use automated verification as the reward signal. Reinforcement Learning with Verifiable Rewards, or RLVR, represents a departure from the previous generation of training methods that relied on human preference judgments. Instead of asking people to rank which answer "sounds better," RLVR uses deterministic checkers, unit tests, and mathematical verification to score model outputs, unlocking reasoning capabilities that earlier training approaches couldn't reliably produce.

What Made the Old Training Method Hit a Wall?

For years, the standard recipe for building large language models was straightforward: pretraining on massive amounts of text, followed by supervised fine-tuning on curated examples, then reinforcement learning from human feedback, or RLHF. Models like GPT-4.5, DeepSeek-V3, and Claude 3.5 Sonnet were built entirely on this approach. By late 2024, however, labs running these models discovered a critical limitation. Scaling up pretraining data and compute power no longer produced equivalent capability gains, particularly on multi-step reasoning tasks like complex math problems or coding challenges.

The bottleneck wasn't raw model size anymore; it was output quality. This forced a reckoning: the field needed a different training paradigm, not just a bigger version of the same one.

Why Does RLHF Struggle With Reasoning Tasks?

Traditional RLHF works by having humans rank multiple candidate responses to the same prompt, then training a separate reward model to predict how humans would score any given output. The policy model, or LLM, is then optimized to maximize that learned reward signal. This approach works well for tone, helpfulness, and safety, but it has a fundamental flaw for reasoning tasks: human raters judge plausibility, confidence, and structure, not necessarily correctness.

A confidently written but mathematically incorrect answer can easily outscore a correct but messier one. The reward model learns to recognize the aesthetic qualities of good answers without actually verifying whether they're right. This is the core limitation that RLVR was designed to solve.

How RLVR Changes the Reward Signal

RLVR replaces the learned reward model with something far more objective: a programmatic checker that verifies correctness directly. The method works differently depending on the task type:

• Code Tasks: The model's output is compiled and run against unit tests, similar to a LeetCode compiler, producing a clear pass or fail signal.
• Math Problems: The final answer is checked against known ground truth, with normalization so that equivalent answers like 1/2 and 0.5 are treated as correct.
• Structured Tasks: Rule-based checkers validate outputs against fixed specifications for grammar, formatting, or compliance.

This deterministic approach eliminates what researchers call "reward hacking." A learned reward model can be gamed by outputs that merely appear good to its learned heuristics. A compiler has no aesthetic preferences; it either passes or fails, regardless of how confident or well-formatted the reasoning chain is.

What Happened When DeepSeek Applied RLVR at Scale?

In January 2025, DeepSeek released three closely related models that demonstrated the power of RLVR. The first, R1-Zero, was trained directly from a base model checkpoint using large-scale RLVR with no supervised fine-tuning step beforehand. The results were striking: reasoning benchmark accuracies climbed substantially, and the model spontaneously developed longer chains of thought and self-verification behavior, none of which were explicitly part of the reward function.

However, R1-Zero also revealed a tradeoff. The model developed serious presentation problems: reasoning chains would mix languages mid-thought, mixing English and Chinese within the same reasoning sequence. General readability was poor, with raw output being difficult to follow even when the final answer was correct. This was the direct consequence of optimizing purely for verification without any guidance on how to present the reasoning clearly.

DeepSeek addressed this limitation with R1, which added a supervised fine-tuning step to teach the model how to format and present its reasoning in a readable way, while retaining the reasoning gains from RLVR. The company also released R1-Distill, demonstrating that the reasoning capability developed through RLVR could be transferred into much smaller models at a fraction of the computational cost.

How to Understand the Three Stages of Modern LLM Training

The evolution of training methods shows a clear progression, each stage addressing limitations of the previous one:

• Supervised Fine-Tuning (SFT): The model learns to imitate high-quality examples from human annotators, but it cannot exceed the skill level of those annotators and has no signal about why one answer is better than another it might generate.
• Reinforcement Learning from Human Feedback (RLHF): The model explores multiple candidate outputs and receives feedback based on human preference judgments, but this feedback often conflates plausibility with correctness, leading to confident wrong answers outscoring messy correct ones.
• Reinforcement Learning with Verifiable Rewards (RLVR): The model receives feedback based on objective verification of correctness, eliminating the gap between what sounds good and what actually works, particularly for reasoning and coding tasks.

Each stage builds on the previous one. RLVR doesn't replace SFT or RLHF entirely; rather, it adds a new layer of optimization that targets the specific challenge of reasoning tasks where correctness can be objectively verified.

Why Does This Matter Beyond Benchmark Scores?

The shift to RLVR represents a fundamental change in how AI labs approach the problem of improving reasoning. For years, the strategy was to scale everything: more data, bigger models, more compute. RLVR shows that the training method itself matters as much as raw scale. By aligning the reward signal with actual correctness rather than human judgment, labs can unlock capabilities that scaling alone couldn't achieve.

This has practical implications for how AI models will be built going forward. Tasks where correctness can be automatically verified, like coding, mathematics, and logic puzzles, can now be trained more effectively. The method also suggests that the next frontier of AI improvement may come not from building bigger models, but from designing smarter training signals that better capture what we actually care about.

The transition from RLHF to RLVR marks the moment when the field moved beyond training models to imitate human judgment and started training them to actually solve problems correctly. That distinction, though subtle, represents a significant step forward in how AI systems learn to reason.