Microsoft researchers have developed a new approach that allows AI systems to learn and improve from their own experiences in real time, without requiring updates to the underlying model. The framework, called EvoLib, transforms raw experiences into an evolving library of reusable knowledge that gets refined over time. This represents a significant shift in how test-time compute, the computational resources spent during inference rather than training, can be converted into measurable performance gains.

What Makes EvoLib Different From Traditional AI Memory?

Most AI memory systems today work like filing cabinets: they store past conversations, reasoning traces, and action histories as static information. EvoLib takes a fundamentally different approach. Instead of simply accumulating more memories, it extracts reusable knowledge from experiences and continually refines that knowledge as new tasks arrive. Think of it like how humans learn: we don't remember every detail of past experiences, but we do remember what matters, such as strategies that work, mistakes to avoid, and skills that transfer across different situations.

The framework operates through three key mechanisms that allow knowledge to evolve and improve over time:

• Consolidation: When new knowledge is extracted from recent experience, EvoLib retrieves similar knowledge from its library and merges them into more general, reusable insights that apply across multiple tasks rather than just one specific situation.
• Weighting mechanism: The system continuously updates how important each piece of knowledge is, based not only on immediate usefulness but also on how much it contributes to generating helpful knowledge for future tasks, allowing the most impactful knowledge to naturally become more prominent.
• Self-supervised learning: EvoLib enables large language models to learn from their own experience during inference without requiring ground-truth labels or external feedback from humans.

How Does EvoLib Convert Compute Into Better Performance?

Researchers tested EvoLib across three challenging task categories: solving mathematical reasoning problems, writing code under efficiency constraints, and making decisions to explore and interact with environments for long-horizon tasks. The results were consistent across all three domains. EvoLib achieved higher performance than both traditional compute scaling methods and strong memory-based learning approaches, and it improved performance more rapidly as the amount of test-time compute increased.

What makes this finding particularly important is that it suggests the path to better AI performance may not simply be storing more memories or spending more compute in isolation. Instead, the greatest gains come from transforming experience into reusable knowledge that can be continually refined and applied across different tasks. This distinction matters because it offers a more efficient way to improve AI systems after they've been deployed, without requiring expensive retraining or model updates.

Does Task Order Matter for Real-World Performance?

One practical concern for deploying such systems is whether they depend heavily on the order in which tasks are encountered. In real-world scenarios, AI systems face diverse types of tasks in arbitrary order, and a useful learning framework needs to be robust to that randomness. Researchers tested EvoLib on the same set of heterogeneous tasks but with different task orderings. The framework consistently improved over existing memory-based learning approaches and maintained stable performance across different orderings.

This finding has significant implications for deployment. It indicates that EvoLib can continually learn from diverse tasks even when they are interleaved, suggesting its practical advantage in real-world scenarios where an agent must handle and learn from a mixed stream of heterogeneous user requests without relying on a structured curriculum.

How to Implement Learning From Experience in AI Systems

• Extract reusable knowledge: Rather than storing raw experiences as static memories, identify and extract generalizable skills and insights from successful solutions and mistakes that can apply to future tasks.
• Consolidate similar knowledge: When new knowledge emerges, merge it with similar existing knowledge to create more general and broadly applicable insights rather than keeping isolated, task-specific observations.
• Continuously reweight knowledge importance: Update how prominently each piece of knowledge is used based on its long-term impact across multiple tasks, not just its immediate utility on the current task.
• Apply to black-box models: Since EvoLib does not require model updates, it can be applied to any language model deployed through APIs, making it accessible without needing to retrain or modify the underlying system.

The implications of this research extend beyond just improving individual AI systems. As AI systems take on longer-running and more complex tasks, the ability to learn from experience becomes increasingly important. The future of AI may depend not only on larger models and more computation, but also on mechanisms that allow systems to continually accumulate, refine, and reuse knowledge.

Microsoft researchers have made their code and experiment results available on GitHub to support future research on memory and knowledge evolution in AI systems. This openness suggests that the broader AI research community is moving toward a model where AI systems can build upon an evolving library of reusable skills and insights after deployment, much like humans do, rather than repeatedly starting from scratch with each new task.