EAGLE 3.1, a new update to a widely-used AI inference acceleration technique, addresses a previously unidentified problem called attention drift that causes language models to lose focus on important context as they generate text faster. The EAGLE team, working with vLLM and TorchSpec teams, discovered that as speculative decoding (a method for speeding up AI model responses) goes deeper, the smaller draft model that predicts tokens starts ignoring the original input and attending to its own generated outputs instead, destabilizing the entire process.

What Is Attention Drift and Why Does It Matter?

Speculative decoding works by having a small, fast draft model propose several tokens ahead of time, while a larger target model verifies all those predictions in a single pass. If the predictions are correct, inference speeds up dramatically. If they're wrong, the system gracefully falls back to the correct tokens. This technique has become one of the most widely deployed acceleration methods in both research and production systems.

However, the researchers found that this approach has a critical weakness. As the draft model makes deeper and deeper predictions, it gradually shifts its attention away from the original input context (called sink tokens) and toward its own previously generated tokens. This self-focus causes the model to become increasingly unreliable, reducing the number of tokens it can correctly predict and destabilizing the overall output quality.

The team identified two underlying causes of this instability. First, the combined input representation becomes imbalanced as higher-layer hidden states dominate the information the draft model receives. Second, the magnitude of hidden states grows across speculation steps because the residual path lacks normalization. Together, these effects make the draft model progressively less stable when trying to predict further ahead.

How Does EAGLE 3.1 Fix the Problem?

EAGLE 3.1 introduces two key architectural improvements designed to keep the draft model stable and focused on the original context. The first is FC normalization, which stabilizes the hidden states after each target model step and before the draft model processes them. Without this normalization, hidden-state magnitude grows unchecked, making the draft model increasingly unreliable. Applying normalization at each step keeps the inputs bounded and predictable.

The second improvement is a post-norm design that feeds normalized hidden states into the next decoding step. This approach makes the system behave more like recursively invoking the draft model across steps, rather than simply stacking additional layers onto the target model. Together, these changes address the root causes of attention drift and restore stability across deeper speculation depths.

Steps to Deploy EAGLE 3.1 in Production Environments

• Integration Method: EAGLE 3.1 lands in vLLM as a config-driven extension, meaning you can enable it by passing configuration parameters without rewriting existing code or retraining models.
• Backward Compatibility: Existing EAGLE 3 checkpoints work directly with EAGLE 3.1 through the same speculative-decoding code path, eliminating the need to retrain or migrate models.
• Training Support: TorchSpec now provides efficient training support for EAGLE 3.1, lowering training overhead and simplifying experimentation workflows for organizations building custom draft models.
• Real-World Example: The research team trained and open-sourced an EAGLE 3.1 draft model for Kimi K2.6, available on HuggingFace, demonstrating how to deploy EAGLE 3.1 with both TorchSpec training and vLLM serving support.

What Performance Improvements Does EAGLE 3.1 Deliver?

The performance gains are substantial across multiple dimensions. In long-context workloads, where models process very long documents or conversations, EAGLE 3.1 achieves up to 2× longer acceptance length compared with EAGLE 3, meaning the draft model can correctly predict twice as many tokens before the target model needs to verify them.

Benchmarks on the Kimi K2.6 model running on Nvidia GB200 hardware show that EAGLE 3.1 delivers 2.03× higher per-user output throughput at low concurrency (one user), meaning responses come back more than twice as fast. As concurrency scales to multiple simultaneous users, the speedup remains meaningful: 1.71× faster at four concurrent users and 1.66× faster at 16 concurrent users.

Beyond raw speed, EAGLE 3.1 demonstrates stronger robustness across real-world deployment scenarios. The update shows better training-time to inference-time extrapolation, meaning models trained on one type of data generalize better to different data at deployment. It also exhibits higher resilience to variations in chat templates, system prompts, and other out-of-distribution inputs that often cause AI systems to degrade in production.

Why Does This Matter for AI Agents and Agentic Workflows?

Faster, more stable inference is particularly important for AI agents, which are autonomous systems that use language models to reason, plan, and take actions. Agents often need to generate multiple reasoning steps, call external tools, and process long context windows to understand complex tasks. Attention drift would cause these agents to lose focus on their original instructions and context, leading to incorrect decisions or tool calls.

By fixing attention drift, EAGLE 3.1 enables agents to maintain coherence and accuracy even when generating longer sequences of reasoning or tool interactions. The 2× improvement in long-context acceptance length is particularly valuable for agents that need to maintain awareness of extended conversation histories or large knowledge bases while making decisions.

EAGLE 3.1 is already merged into vLLM main and will ship in version 0.22.0, making it immediately available to organizations building and deploying AI agents at scale.