A new framework called Hybrid Open-Ended Tri-Evolution (HOTE) enables AI agents to autonomously improve their research capabilities by learning from themselves rather than relying on human feedback or fixed training data. The system uses three co-evolving components, proposer, solver, and judge, that work together through reinforcement learning to tackle complex, open-ended research questions. An 8-billion-parameter model trained with HOTE surpassed larger open-source models ranging from 8 to 32 billion parameters on three deep research benchmarks.

What Makes This Different From How AI Models Are Usually Trained?

Traditionally, AI models learn from human feedback or fixed datasets. Reinforcement Learning from Models (RLM) represents a shift in how AI systems improve themselves. Instead of humans evaluating every response, an AI reward model scores outputs, and the language model being trained updates itself to produce higher-scoring responses over time. This approach powers modern AI systems like Claude, ChatGPT, and Gemini, but it has documented limitations when applied to open-ended tasks like research report generation, where there is no single correct answer.

The HOTE framework addresses this gap by introducing a judge module that dynamically generates evaluation rubrics by comparing multiple responses to the same query. This prevents the system from gaming the reward signal, a problem known as reward hacking, while maintaining the difficulty and learnability of research tasks.

How Does the Three-Part Agent System Actually Work?

• Proposer Module: Performs information seeking based on weaknesses identified by the judge and generates challenging yet learnable research queries that push the solver to improve.
• Solver Module: Receives a research query, generates a plan, conducts multi-turn information seeking through web search, integrates findings, and produces a referenced research report with citations.
• Judge Module: Dynamically creates evaluation rubrics by analyzing the strengths and weaknesses of multiple solver responses, then assigns reward scores to guide the solver's improvement without relying on predetermined correct answers.

The three modules engage in what researchers call a game, where the proposer continuously tries to find queries that expose the solver's weaknesses, while the solver improves its research capabilities in response. The judge ensures this process remains fair and prevents the solver from simply learning to fool the reward system.

Why Does This Matter for AI Development?

Deep research, which emphasizes autonomous handling of open-ended, long-cycle, and highly complex information retrieval and integration, has become a critical task for AI agents in real-world applications and a step toward artificial general intelligence. Closed-source systems from OpenAI, Anthropic, and other labs have demonstrated near-human research capabilities, but the open-source community has struggled to build comparable systems that can autonomously plan workflows and improve without massive human annotation efforts.

The HOTE framework bridges two critical paths toward more capable AI: deep research and agent evolution. Previous agent evolution systems were limited to tasks with verifiable answers, like mathematics or coding problems. HOTE extends this capability to open-ended tasks where there is no single correct answer, making it applicable to real-world research, report generation, and complex problem-solving.

What Do the Experimental Results Show?

The research team tested HOTE on three long-form deep research benchmarks. An 8-billion-parameter model trained with HOTE outperformed the strongest open-source models in the 8 to 32 billion parameter range, as well as models trained using state-of-the-art deep research training methods, while requiring less training time overhead. The framework also demonstrated that the evolution of all three modules, proposer, solver, and judge, is essential; removing any one of them degraded performance.

The researchers also introduced a dual-mode hybrid training strategy that includes both tool-use and no-tool modes. This approach achieves mutual benefit between the two modes and significantly improves training efficiency. The system effectively maintains the difficulty of synthetic queries during training, preventing the solver from becoming overconfident on easy tasks.

What Are the Broader Implications for AI Reward Systems?

Reinforcement Learning from Models has become the foundation for training frontier AI systems, but it faces three documented failure modes: reward hacking, where models learn to exploit the reward system rather than genuinely improve; the circularity problem, where the reward model's own biases become embedded in the trained model; and model collapse, where the system stops improving or degrades over time.

HOTE's dynamic judge module directly addresses reward hacking by continuously evolving evaluation criteria based on the solver's actual performance. This prevents the solver from finding shortcuts or exploiting static reward signals. The framework demonstrates that agentic AI, where multiple specialized components work together, may be essential for building AI systems that can improve themselves reliably in open-ended environments.

The shift from Reinforcement Learning from Human Feedback (RLHF) to Reinforcement Learning from AI Feedback (RLAIF) to the more mature Reinforcement Learning from Models represents a fundamental change in how AI systems scale. Human annotation is slow, expensive, and inconsistent; AI-based evaluation enables billions of comparisons that would be impossible for human raters to complete. HOTE extends this capability to domains where traditional reward models struggle, opening new possibilities for autonomous AI agent development.