As large language models (LLMs) evolve from solo tools into coordinated teams, a fundamental challenge has emerged: how do you train AI systems to work together effectively? A comprehensive new survey published on arXiv maps the emerging field of multi-agent reinforcement learning (MARL) for LLM-based systems, revealing the technical frameworks that companies like Moonshot, OpenAI, and Anthropic are using to scale AI collaboration from a handful of agents to hundreds working in parallel.

The research introduces "orchestration traces" as a unifying concept for understanding how AI teams coordinate. Rather than optimizing individual agent actions alone, orchestration traces capture the entire temporal interaction graph: when agents are spawned, whom work is delegated to, how agents communicate, which tools they use, how outputs are aggregated, and when the process stops. This shift from single-agent to multi-agent thinking represents a fundamental change in how AI systems are trained.

What Are Orchestration Traces and Why Do They Matter?

Orchestration traces function as a common language for auditing reward design, credit assignment, and learning in multi-agent systems. Think of them as detailed blueprints of how a team of AI agents collaborates on a task. The survey identifies three critical technical axes that govern this collaboration: reward design, credit assignment, and orchestration learning.

The timing of this research is significant. As of May 2026, the field has reached a inflection point where academic methods are finally catching up to industrial deployments. Moonshot's Kimi K2.5 agent swarm, trained with Parallel-Agent Reinforcement Learning (PARL), scales to 100 sub-agents and 1,500 coordinated steps or tool calls. The newer K2.6 version expanded this to 300 sub-agents and 4,000 coordinated steps, adding cross-vendor coordination capabilities. These aren't theoretical exercises; they represent real systems handling complex, multi-step workflows.

How Do AI Teams Learn to Coordinate Effectively?

• Reward Design: The survey identifies eight distinct families of rewards that govern multi-agent behavior, including system-level properties such as parallelism speedup, split correctness, and aggregation quality. These rewards tell agents whether they're succeeding at the team level, not just individual task completion.
• Credit Assignment: Eight different credit-bearing units exist across the spectrum from individual tokens to entire teams. Agent-level, role-level, turn-level, and orchestrator-level signals are beginning to fill gaps, though explicit counterfactual message-level credit remains sparse in current implementations.
• Orchestration Learning: Five sub-decisions govern how teams operate: when to spawn new agents, whom to delegate work to, how agents should communicate, how to aggregate partial outputs, and when to stop the process. Notably, as of May 2026, no explicit reinforcement learning training method exists for the stopping decision.

The research pool analyzed by the survey includes over 8,400 tagged papers and 3,232 exclusion records, providing an unprecedented view of the landscape. Academic methods have produced systematic multi-agent reward fine-tuning (RFT) paradigms, hierarchical group reward policy optimization (GRPO) decompositions for LLM teams, and credit-assignment methods targeting message-level counterfactuals and Shapley-based agent-level credit.

What's the Gap Between Academic Research and Industrial Deployment?

A significant divide exists between what researchers publish and what companies deploy at scale. OpenAI's Codex application functions as a command center managing parallel software-engineering agents, while Anthropic's Claude Code ships with built-in and user-defined sub-agents. An Anthropic engineering post-mortem documented sixteen parallel Claude instances jointly building a C compiler, demonstrating the practical power of coordinated AI teams.

However, these industrial systems primarily document deployment shape and engineering constraints rather than disclosing whether multi-agent coordination itself is an explicit reinforcement learning training target. Kimi represents the clearest public example of a trained multi-agent orchestration system, while Codex and Claude Code mainly reveal deployment architecture and operational boundaries.

The survey connects academic methods to this public industrial evidence, treating the scale gap not as independent verification of industrial training traces but as a gap between publicly reported deployment envelopes and open academic evaluation regimes. This distinction matters because it shows where the field is heading: toward systems that explicitly learn how to orchestrate teams, not just how to perform individual tasks.

The research community has released artifacts to support continued progress, including an 8,484-entry tagged paper pool, a 3,232-record exclusion log, scripted corpus statistics, and a minimal JSON schema for replayable orchestration traces. The survey closes with fifteen research directions spanning algorithms, rewards, systems, safety, and evaluation, signaling that multi-agent reinforcement learning for LLMs remains an active frontier with substantial open problems.

As AI systems become more capable, the ability to coordinate multiple agents efficiently will likely become as important as training individual models. The frameworks emerging from this research suggest that the next generation of AI breakthroughs may come not from larger single models, but from smarter teams of smaller ones.