A team at Bridgewater AIA Labs and the University of Illinois has demonstrated that the quality of reward signals used to train AI models is far more fragile than previously understood, and that data corruption can completely undermine reinforcement learning with verifiable rewards (RLVR), a technique increasingly used by frontier labs to improve reasoning. The researchers fine-tuned a model called ReViSQL-K2.6 that now exceeds human accuracy on text-to-SQL tasks, a real-world benchmark where AI systems translate natural language questions into database queries. But the path to success revealed something troubling: nearly one-third of the "correct" training signals were actually reinforcing wrong answers.

Why Are Reward Signals So Vulnerable to Corruption?

In reinforcement learning, a model learns by receiving a numerical reward signal for each action it takes. If that signal is wrong, the model learns the wrong behavior. Unlike supervised learning, where a human can review a batch of labeled examples, RLVR relies on a single scalar reward as the entire learning signal for each training step. A mislabeled instance does not just add noise; it reverses the signal entirely, degrading learning significantly.

The Bridgewater team audited 2,500 examples from BIRD Train, a widely-used text-to-SQL training dataset, and found errors in every component: the natural language questions, the external knowledge provided, and in more than half of the "golden SQL queries" that models are supposed to match. When they cleaned up the evaluation set, BIRD Mini-Dev, they discovered annotation errors in 52.8% of instances.

This discovery matters because RLVR is becoming central to how frontier labs train their most capable models. As one observer noted during a recent industry discussion, the reinforcement-learning environments that labs purchase from third-party vendors are often "rushed and vibe coded," and the reward signals themselves may not be pure enough to reliably train models.

How Did Researchers Fix the Reward Signal Problem?

• Expert-Verified Training Data: The team created BIRD-Platinum, a cleaned-up training set where human experts and an AI auditor (OpenAI's o3) reviewed each instance. The AI auditor achieved 90.6% precision but only caught 24.5% of the errors that human experts identified, demonstrating that human review was essential to catch the majority of annotation errors.
• Semantic Equivalence Testing: Standard reward functions check whether a generated SQL query returns the same result as the correct query on a single database instance. But this misses errors like wrong join keys or dropped predicates that only show up on different data. The researchers used a tool called VeriEQL to test for true semantic equivalence, and found that 32.8% of positive rewards were given to queries that were not actually equivalent to the correct answer.
• Downweighting Unreliable Signals: When VeriEQL detected that a query might not be semantically equivalent, the team reduced the strength of the reward signal rather than treating it as a full positive example.

The results were dramatic. Training on BIRD-Platinum alone lifted ReViSQL-K2.6 to 88.55% accuracy on the cleaned evaluation set. When the team fine-tuned a different model, Qwen3-235B, on the verified data, accuracy improved by 12 to 16 percentage points across multiple benchmarks and SQL dialects compared to training on the original noisy dataset.

What Does This Mean for Frontier AI Labs?

The ReViSQL-K2.6 model now exceeds the human benchmark of 92.96% accuracy when selecting from 16 candidate queries, and does so at a cost of $0.56 per task. This is 12 to 15 times cheaper than frontier models like Claude Fable 5 and GPT-5.6 Sol Ultra, which score in the mid-80s but cost far more to run at scale.

However, the broader implication is more concerning. Industry observers have raised alarms about the supply chain of reinforcement-learning environments. One person who previously worked inside a vendor described the work as "rushed and vibe coded," and there are reports of labelers prompt-injecting their employers so that AI systems would do the labeling work for them. If the reward signals themselves are compromised, then models trained on those signals will inherit those flaws, potentially at scale.

Industry

"I would be not doing my job if I didn't say this does give me some real qualms about recursive self-improvement as a strategy," Nathan Labenz stated, raising concerns about what happens when models trained on impure signals are then used to train the next generation of models.

Nathan Labenz, Founder, Cognitive Revolution

The Bridgewater team's solution, while effective, required significant manual effort: expert review, conflict resolution loops, and a second pass of cleanup by a different team. This approach does not scale easily to the thousands of tasks and environments that frontier labs are now using to train their most advanced reasoning models.

Steps to Protect AI Training Pipelines From Corrupted Reward Signals

• Audit Representative Training Samples: Before committing to large-scale RLVR training runs, systematically review a representative sample of training data. The Bridgewater team's finding that 52.8% of evaluation examples contained errors suggests that spot-checking is insufficient; comprehensive review is necessary.
• Test for Semantic Equivalence: Consider whether the reward function actually captures the behavior you want to train. Result-based rewards are convenient but incomplete. Testing for semantic equivalence, as the Bridgewater team did with VeriEQL, adds computational cost but catches errors that simpler metrics miss.
• Scrutinize Third-Party RL Environments: Be skeptical of third-party reinforcement-learning environments. One industry observer has proposed that labs publish representative samples of their RL environments for outside review, similar to how open-source projects invite code audits. This transparency could help catch "rushed and vibe coded" work before it corrupts training at scale.

The code, data, and training recipes for ReViSQL are publicly available, and the cleaned datasets (BIRD-Platinum and Arcwise-Plat-SQL) have been released to the community. This transparency allows other labs to benefit from the data-cleaning work and to replicate the approach on their own domain-specific tasks.

As frontier labs scale up recursive self-improvement, where models train the next generation of models, the purity of the reward signal becomes a critical dependency. The Bridgewater research shows that this dependency is fragile, and that even widely-used public datasets contain enough errors to significantly degrade learning. For labs building on top of third-party RL vendors, the risk is even higher.