AI agents are creating an observability crisis that most enterprises haven't anticipated yet. While token prices continue to drop, the sheer volume of logs, traces, and tool-call chains that agents generate is overwhelming monitoring pipelines and inflating costs across organizations. The problem isn't just technical; it's forcing companies to rethink how they collect, store, and analyze telemetry data at scale.

Why Are AI Agents Generating So Much Telemetry?

The explosion in agentic software development has fundamentally changed how much data flows through enterprise systems. Agents running in loops, making repeated calls to language models, and executing complex reasoning chains leave behind detailed records of every decision and action. At companies like Cribl, which specializes in high-throughput telemetry processing, the volume is staggering. Some customers generate tens of terabytes of logs every single hour, and agents amplify that problem significantly.

The issue becomes acute when agents are left to optimize for vague objectives. In one real-world example, a research team at Cribl deployed an agent tasked with improving model performance by "thinking more." The agent interpreted this literally, making increasingly expensive calls to large language models (LLMs) overnight. By morning, the token bill had ballooned to an enormous figure, shocking the researchers who discovered it. The agent had no built-in guardrails to question whether each additional computation was worth the cost.

"Token costs keep dropping compared to a few years ago, but token use is going up quite a bit. Models today and use cases powered by LLMs are much more complex, longer running, and tend to consume a lot more tokens than use cases from 2022 and 2023," explained Nikhil Mungel, Head of AI R&D at Cribl.

Nikhil Mungel, Head of AI R&D at Cribl

How to Control Agent Costs and Observability Overhead

• Implement Token Budgets: Set hard limits on how many tokens a single agent experiment or workflow can consume before it pauses and alerts a human operator for approval, preventing runaway costs.
• Standardize Telemetry Schemas: Use frameworks like the Open Cybersecurity Schema Framework (OCSF) to standardize what agents emit, making it easier to filter, compress, and route logs efficiently without losing critical information.
• Distinguish Internal vs. Customer-Facing Spend: Separate the token consumption of internal agentic tools (like code generation and research automation) from the tokens agents burn while serving external customers, since the cost drivers and optimization strategies differ significantly.
• Monitor Objective Functions Carefully: Define agent goals with precision and include explicit constraints on computational effort, not just outcome targets, to prevent agents from optimizing in unexpected and expensive directions.

The Disconnect Between Token Prices and Total Spend?

One of the industry's most confusing paradoxes is that token prices have fallen dramatically while total spending on AI has climbed. This happens because the complexity and length of agentic workflows have increased far faster than prices have decreased. A single agent making dozens of sequential calls to an LLM, each processing thousands of tokens, can easily cost more than a simpler, single-shot query from years past, even at lower per-token rates.

The problem is compounded by the fact that teams often reach for the most capable (and expensive) models by default. Without clear cost visibility or governance, engineers tend to use premium models like GPT-4 or Claude when smaller, cheaper alternatives might suffice for many tasks. This behavior, multiplied across hundreds of agents in an enterprise, creates a cost spiral that's difficult to reverse.

What Role Does Agent Governance Play?

The broader challenge extends beyond just telemetry costs. As enterprises move from managing a handful of agents to managing dozens or hundreds built on different frameworks and connected to different tools, the operational complexity becomes unmanageable. This shift has sparked intense interest in "agent control planes," a concept borrowed from telecommunications infrastructure that applies governance, security policies, and operational visibility across all agents in an environment.

An agent control plane determines which agents can act, what systems they can access, how they interact with other agents, and how their behavior is monitored. However, buying the technology is only half the battle. Organizations still need to decide what policies to enforce, who owns those policies, and how accountability works when agents make decisions or take actions.

"Buying a control plane is not the same thing as having a governance strategy. The technology can enforce policies, but organizations still need to decide what those policies are, who owns them and how accountability works when agents make decisions or take actions," noted Amanda Downie, Staff Editor at IBM Think.

Amanda Downie, Staff Editor at IBM Think

The governance challenge is ongoing and operationalized, not a one-time implementation. As agents continue to proliferate, the need for standardized schemas, cost controls, and clear decision-making frameworks becomes more urgent. Without these guardrails in place, the observability bill will keep climbing, and the risk of silent failures will grow.