Enterprise software companies are discovering that plummeting AI model prices don't automatically translate into healthier profit margins. While inference costs have dropped dramatically, the way companies are actually using AI has fundamentally changed the economics. A single user request now triggers dozens of billable operations instead of one, creating what industry experts call the "100x problem".

What Is the 100x Problem in AI Pricing?

The shift from simple chatbots to complex AI agents has upended the traditional software pricing model. A traditional chatbot turns one user question into one model call. An agent system, by contrast, breaks down that same request into a chain of planning, retrieval, tool use, verification, summarization, and follow-up decisions. The user sees one answer. The vendor pays for the entire loop.

Consider a practical example: A customer support agent answering "What did our top customer ask about last week?" typically requires seven separate priced operations before returning an answer. This includes the initial user prompt, system instructions, database retrieval, multiple model calls for tool selection and summarization, and follow-up decision-making. The result is roughly 35,000 input tokens billed for what appears to be a simple query, costing between $0.10 and $0.40 per request on a frontier model. Scale that to a million queries per month, and the monthly bill reaches six figures.

The problem compounds because the customers receiving the most value from the product are often the ones generating the highest inference costs. A power user running 50 agent invocations daily on a $40-per-seat subscription plan can cost more in inference than the plan charges, creating negative gross margins on the vendor's most engaged customers.

How Are Companies Controlling Runaway AI Costs?

Industry experts and vendors are converging on several technical approaches to manage token amplification without sacrificing functionality. These strategies are not new, but they have become critical for survival in the current AI landscape.

• Cost-Aware Routing: A small classifier model decides which tier of AI model (equivalent to Haiku, Sonnet, or Opus) handles each query, cutting inference bills by around 60% without degrading quality.
• Prompt Caching: Anthropic, OpenAI, and Google now offer 75 to 90% discounts on cached prefixes, reducing redundant processing of repeated instructions.
• Context Discipline: Vendors can truncate tool outputs, prune reasoning traces, and cap tool depth to prevent agents from consuming unnecessary tokens.
• Speculative Decoding: For self-hosted deployments, this technique guarantees 2 to 3 times effective throughput on the same computing hardware.

However, implementing these solutions requires a fundamental shift in how companies think about infrastructure. The companies building this layer well are starting to resemble financial trading systems, where every routing decision carries a price tag, every path has its own profit-and-loss statement, and every customer operates on a metered budget.

What Strategic Changes Do Vendors Need to Make?

Four critical moves separate companies that will maintain healthy margins in the next 24 months from those that won't. First, inference cost must become a first-class metric, tracked per-feature, per-tenant, and per-query class the same way cloud costs were tracked starting in the mid-2010s. Second, teams need to budget like media buyers, setting cost-per-thousand-queries ceilings per feature and alerting on overruns. Engineering will not enforce cost discipline on its own.

Third, the router that directs queries to appropriate model tiers should be treated as core infrastructure, not merely an optimization. It is the new load balancer. Fourth, companies must audit their prompts quarterly. A 4,000-token system prompt that grew organically over six months represents a six-figure bill in slow motion, and most teams have never read their own production prompts end to end.

"For my team, the cost of compute is far beyond the costs of the employees," said Bryan Catanzaro, VP of Applied Deep Learning at Nvidia.

Bryan Catanzaro, VP of Applied Deep Learning, Nvidia

The strategic implication extends beyond the observation that AI is expensive. The dominant business model assumed by most AI-native company plans does not survive contact with agentic workloads. Salesforce's Agentforce, for example, has faced public scrutiny over a widening gap between marketing demonstrations and capabilities actually shipped to customers, a gap that typically opens when promised functionality is technically possible but economically unviable to serve at the price the subscription plan implies.

The visible symptoms have begun leaking into public coverage. As more enterprises adopt AI agents, the correlation between agent adoption and gross margin contraction has moved from a theoretical risk to a primary profit-and-loss headwind. Several vendors are now privately reporting negative gross margins on heavy users, mirroring recent cloud expenditure reports from the Bessemer "Supernova" cohort of well-funded startups.

The takeaway for enterprise software companies is clear: cheaper models help, but they do not fix a product architecture that turns one user-visible prompt into dozens of billable operations. Without deliberate cost management, the race to deploy AI agents faster than competitors may inadvertently race companies toward unprofitable growth.