Human forecasters remain ahead of AI bots in competitive forecasting tournaments, but the gap is narrowing rapidly as frontier reasoning models like OpenAI o3 emerge as the primary driver of bot performance. A comprehensive analysis of 11 forecasting studies conducted between October 2024 and May 2026 reveals that the choice of underlying AI model matters far more than sophisticated prompting techniques or search strategies.

How Are AI Bots Actually Competing Against Human Forecasters?

On the Metaculus forecasting platform, which runs quarterly tournaments where bots and professional human forecasters compete on the same questions, the results tell a clear story. In the first quarter of 2025, professional forecasters beat bot teams by 17.7 points on a peer-score metric, with statistical significance over 99% confidence. By the second quarter of 2025, that advantage had grown to 20 points. Yet within those same tournaments, individual bots using frontier models placed remarkably well. OpenAI o1 reached 25th place out of 617 competitors in the Metaculus Cup, while OpenAI o3 emerged as the single best large language model at single-prompt scaffolding, a technique that structures how the model approaches a problem.

The picture becomes more nuanced when comparing bots to the general public. According to ForecastBench, a forecasting benchmark unaffiliated with Metaculus, the median public forecaster ranked second only to superforecasters a year ago. By October 2025, that same median public forecaster had dropped to 22nd place, while AI models like GPT-4.5 now forecast better than the average person. However, superforecasters still maintain a meaningful edge, with a Brier score of 0.081 compared to 0.101 for the best AI models, a difference that translates to roughly 20% better accuracy.

What Makes Frontier Models So Much Better Than Everything Else?

The research reveals a striking finding: in the first four quarters of the FutureEval leaderboard, the single biggest predictor of bot performance was whether the bot used a frontier model at all. Simple one-shot bots, meaning bots that asked the model a question once without complex follow-up techniques, placed in the top five on the leaderboard if they used a frontier model. By fall 2025, this lesson had become obvious to the bot-building community. Of 39 bot makers surveyed, 34 used a frontier model. Only after that shift did more sophisticated techniques like prompt engineering and research aggregation start showing meaningful performance differences.

This finding upends conventional wisdom in AI development. Engineers often assume that clever prompting, fine-tuning, and complex system design are the keys to better performance. The forecasting data suggests otherwise. A frontier model is worth roughly nine months of base model progress in terms of performance gains. In the fall 2025 tournament, the top five scaffolded bots beat their non-scaffolded baselines by 5 to 11 points per question. Frontier base models improve at about 0.9 points per month, making good scaffolding equivalent to nine months of waiting for the next generation of models to arrive.

What Techniques Actually Work Once You Have a Frontier Model?

After establishing that frontier models matter most, the research identified several techniques that do provide meaningful improvements:

• Ensemble Aggregation: Eighty-six percent of fall 2025 tournament winners combined multiple forecasts from different models or prompting strategies. Most external research reviewed also converged on this practice, suggesting that combining predictions reduces the risk of any single model's biases.
• Calibration Adjustment: A technique called Platt scaling, which adjusts predictions after the fact to match historical accuracy patterns, improved bot Brier scores by 0.016 on binary questions and 0.005 on multiple-choice questions with over 99% statistical confidence. This simple mathematical adjustment was worth more than many complex prompting innovations.
• Prediction Capping: Setting maximum and minimum bounds on predictions emerged as the strongest differentiator among tournament winners, with a correlation of 0.48 and statistical significance at the 99.5% confidence level. This prevents models from making extreme predictions that, while occasionally correct, hurt overall accuracy.
• Research Breadth: The number of different research sources a bot consulted correlated with tournament performance at a rate of 0.42, with 99% statistical confidence. However, no single search provider gave a significant advantage over others, suggesting that breadth matters more than the specific choice of tool.

Interestingly, automated prompt engineering, a technique that uses algorithms to optimize how questions are phrased, showed mixed results. When researchers tested it on GPT-4.1 and GPT-4.1-nano models, they saw large gains on the nano model but moderate gains on GPT-4.1 and no gains on DeepSeek-R1. When extended to OpenAI o3 in a follow-up study, the technique showed no significant improvement at all.

How Do Bots Perform in Real Tournament Settings?

Beyond the Metaculus platform's controlled comparisons, bots have shown competitive performance in actual forecasting tournaments. In the summer 2025 Metaculus Cup, a bot using Gemini 2.5 Pro with news aggregation placed 37th out of 551 total forecasters, putting it in the top 6% of all competitors. Among forecasters who participated in more than 75% of questions, it ranked 37th out of 51, placing it in the top 72% of engaged forecasters. In the fall 2025 Cup, the same bot configuration ranked 23rd out of 539 humans, or top 5%, and 23rd out of 42 high-participation forecasters, placing it in the top 54%. By spring 2026, a bot using Claude 4.5 Sonnet ranked 33rd out of 1,130 humans, or top 3%, and 33rd out of 54 high-participation forecasters, placing it in the top 61%.

These results show that bots are competitive with active human forecasters but still trail the very best. The gap appears to be closing, but human forecasters, particularly those with experience and engagement, retain an advantage.

What Does This Mean for the Future of AI Reasoning Models?

The forecasting research provides a real-world test of how well frontier reasoning models actually perform on complex, open-ended questions. Unlike benchmark tests that measure performance on fixed datasets, forecasting tournaments measure live prediction accuracy on questions that have not yet been resolved. This makes them a more rigorous test of genuine reasoning capability.

The dominance of frontier models in forecasting performance aligns with broader trends in AI development. Between 2024 and 2026, reinforcement learning emerged as the primary method for teaching AI systems to reason. OpenAI's o1 and o3 models, DeepSeek-R1, Anthropic's extended-thinking Claude models, and Google's Gemini 2.5 Thinking all use reinforcement learning as the core of their post-training pipeline. This technique, which teaches models to learn through trial and error guided by reward signals, mirrors the approach that powered AlphaGo's breakthrough in 2016.

The forecasting data suggests that this investment in reasoning capability is paying off. Frontier models are not just incrementally better than previous generations; they represent a qualitative shift in how AI systems approach complex problems. For organizations building AI applications, the lesson is clear: the choice of base model matters more than engineering sophistication. Investing in access to frontier models should be a higher priority than optimizing prompts or building complex retrieval systems.