Healthcare AI Systems Are Sprawling Out of Control, and Regulators Aren't Ready
Healthcare organizations deploying interconnected AI agents face a critical governance blind spot: existing regulatory frameworks were designed for single AI models, not the complex webs of autonomous agents now operating in clinical workflows. A new arXiv research paper warns that this mismatch is creating what researchers call "agent sprawl," where AI agents proliferate across hospitals and health systems without clear ownership, permission boundaries, or decommissioning plans.
What Is Agent Sprawl and Why Should Healthcare Leaders Care?
Agent sprawl refers to the uncontrolled proliferation of AI agents across an organization without clear accountability or retirement procedures. In healthcare, this problem is particularly acute because failures do not occur in isolation. When one AI agent hands off a task to another, or when multiple agents interact to support a clinical decision, the risk emerges from the interaction chain itself, not from any single component. This means that if something goes wrong, it may be impossible to trace the error back to a single accountable system or owner.
The problem compounds existing regulatory exposure. Healthcare organizations already operate under strict compliance requirements, including the California Health Care Services AI Act Disclosure Requirements, which assume that AI decision points are identifiable and discrete. Multi-agent systems shatter that assumption. A single compromised or misbehaving agent can propagate errors or unauthorized actions across an entire deployment without triggering existing monitoring thresholds.
How Can Healthcare Organizations Govern Multi-Agent AI Systems?
The arXiv research proposes a structured framework with specific governance requirements that healthcare compliance teams should implement immediately:
- Named Ownership: Each deployed AI agent must be assigned to a specific accountable owner within the organization, eliminating ambiguity about who is responsible for the agent's behavior and performance.
- Explicit Permission Boundaries: Permissions must be formally approved and auditable for each agent, rather than inherited or defaulted, ensuring that agents cannot access data or systems beyond their intended scope.
- Continuous Monitoring Across Interactions: Audit logs must capture interactions between agents, not just individual agent outputs, so that error chains can be traced across the entire system.
- Formal Retirement Criteria: Each agent must have documented triggers for decommissioning, such as model version changes, data access expiration, or clinical guideline updates, preventing outdated agents from operating indefinitely.
- Multi-Agent Risk Classification: Organizations must distinguish between single-model deployments and multi-agent systems during intake and approval workflows, requiring separate governance review for any multi-agent configuration.
The framework directly addresses a gap that the MIT Sloan analysis of authority gaps in agentic AI identified more broadly, but this research extends those findings into the high-stakes context of regulated healthcare operations where patient safety and compliance obligations are paramount.
What Are Regulators Watching For?
Healthcare compliance teams should monitor several regulatory signals. The FDA AI/ML Software as Medical Device Guidance may be updated to address multi-agent configurations explicitly, as the current framework does not map cleanly onto distributed accountability models. The International Telecommunication Union's 2025 AI Governance Report identified AI agents as a central governance challenge requiring new frameworks for traceability and multi-agent coordination, serving as an authoritative benchmark for enterprise compliance programs.
The Cloud Security Alliance documented ten real AI agent security incidents between January 29 and March 18, 2026, spanning poisoned agent registries, prompt injection attacks on developer tooling, and unauthorized infrastructure diversion by autonomous agents. The report concluded that enterprise teams lack foundational controls for safe agentic deployment, specifically citing failures in identity binding, audit log integrity, and shadow traffic detection.
The IMDA Model AI Governance Framework for Agentic AI offers the closest existing policy analog to the ownership and permission-bounding requirements proposed in the arXiv research. Any forthcoming updates to that framework may signal the direction of binding healthcare-specific rules. Organizations should also watch for enforcement actions or audit findings in the Centers for Medicare and Medicaid Services (CMS) and state health agency contexts that target agent sprawl or undocumented AI handoff chains, as regulators are increasingly scrutinizing AI-assisted clinical decision workflows.
Why This Matters Beyond Healthcare
While this research focuses on healthcare, the governance gaps it identifies extend across industries deploying multi-agent AI systems. Microsoft has publicly positioned governance, specifically identity verification, policy enforcement, and human oversight, as mandatory preconditions for deploying AI agents in enterprise environments, placing these requirements ahead of raw model capability. This stance elevates governance from a post-deployment concern to a gate that must be cleared before any agentic AI system goes into production.
The timing is critical. As organizations worldwide accelerate AI adoption, the gap between deployment speed and governance maturity is widening. Healthcare organizations that audit their current AI systems, assign named owners to each agent, document permission boundaries, and establish formal retirement criteria will be better positioned to meet emerging regulatory requirements and avoid costly compliance failures.