A new technique called Agentic ESOpt could solve one of the biggest bottlenecks in building smarter AI agents: the massive amount of computing power needed to train them. Instead of using the standard reinforcement learning approach that demands enormous GPU memory, researchers from the National University of Singapore, Southern University of Science and Technology, and Oxford propose using evolution strategies, a fundamentally different optimization method that requires only inference-level GPU memory.

Why Is Training AI Agents So Computationally Expensive?

When AI labs train large language models (LLMs) to act as agents, they face a unique challenge. Unlike single-turn tasks where a model generates one response, agents must make decisions across many steps, using tools, processing long contexts, and coordinating multi-step workflows. This creates what researchers call "long-horizon" reasoning, where the model must plan and adapt across dozens or hundreds of interactions.

Traditional reinforcement learning methods struggle with this problem. They require storing massive amounts of data during backpropagation, the process where the model learns from its mistakes. As trajectories get longer and more complex, these memory demands become prohibitive. Even state-of-the-art models like Qwen3, DeepSeek-R1, and Gemini 2.5 still struggle with uncommon tool APIs and specialized tasks, making efficient fine-tuning essential.

How Does Agentic ESOpt Work Differently?

Evolution strategies take a radically different approach. Instead of backpropagating through every decision the agent makes, the method samples small random variations of the model's parameters, tests how well each variation performs, and then updates the parameters based on which variations worked best. Think of it like testing thousands of slightly different versions of the agent and keeping the improvements from the best performers.

The key advantage is that this forward-only process requires storing only the random noise seed used to create variations, not the full computational history. This means the GPU memory footprint stays at inference levels, the same amount needed to simply run the model. The researchers demonstrated that this approach enables full-parameter optimization of a 27-billion parameter Qwen3.5 model on just four H100 GPUs, a significant reduction in hardware requirements.

What Results Did the Researchers Achieve?

The experimental results suggest evolution strategies may be structurally superior to reinforcement learning for long-horizon tasks. On a long-horizon Sudoku puzzle task, Agentic ESOpt outperformed reinforcement learning methods by 12.50 percent when using the smaller Qwen3.5-4B model. On a more complex web automation benchmark called WebArena-Lite, full-parameter optimization of the larger Qwen3.5-27B model improved the baseline performance by 6.69 percent, and when combined with an additional technique called Trace2Skill, it achieved a 2.42 percent improvement over that baseline.

The method also supports test-time compute, where the model spends additional computational resources during inference to reason through harder problems. In automatic heuristic design experiments, Agentic ESOpt improved its matched baseline in 28 of 36 test settings, suggesting the approach generalizes across different problem types.

How to Implement Evolution Strategies for Agent Training

• Sample Parameter Perturbations: Generate random variations around the current model parameters using Gaussian noise, then evaluate how well each perturbed version performs on the task.
• Collect Trajectory-Level Rewards: Rather than assigning credit to individual actions, evaluate the entire sequence of decisions the agent makes and assign a single reward score to the complete trajectory.
• Apply Reward-Weighted Updates: Update the model parameters in the direction of variations that produced higher rewards, using a cosine decay schedule to balance exploration of new parameter regions with refinement of good solutions.
• Compose with Prompt Optimization: Combine parameter updates with prompt-space optimization, allowing the model to adapt both its internal weights and its instructions simultaneously for maximum flexibility.

This approach addresses a critical limitation of traditional reinforcement learning for agents. As task horizons grow longer, credit assignment becomes harder because the model must figure out which of its many decisions led to success or failure. Evolution strategies sidestep this problem by treating the entire trajectory as a unit, making them more scalable as reasoning chains get more complex.

What Does This Mean for the Future of AI Agents?

The research signals an important shift in how AI labs might approach agent training. For years, the assumption was that evolution strategies were cheaper but slightly weaker than reinforcement learning. Agentic ESOpt suggests that for long-horizon reasoning, evolution strategies may actually be the better choice structurally, not just economically. This could democratize agent development by making it feasible for labs with more modest hardware budgets to fine-tune large models for specialized tasks.

The work also aligns with a broader pattern emerging in AI research. As noted by AI research analyst Scott Loftesness, the frontier of AI development is shifting from making individual models smarter to building better systems around those models. Test-time compute, adaptive reasoning, memory systems, and agent frameworks are becoming as important as raw model capability. Agentic ESOpt exemplifies this trend by treating the model as one component within a larger optimization loop.

The implications extend beyond just cost savings. By making agent training more accessible, this technique could accelerate development of AI systems for specialized domains like scientific research, software engineering, and cybersecurity, where long-horizon reasoning and tool use are essential. The fact that the method works well with test-time compute suggests it could become a standard building block in next-generation AI agent architectures.