Cloudflare has unveiled a suite of infrastructure innovations that make it significantly cheaper and faster to run massive AI language models like Moonshot AI's Kimi K2.5, which contains over 1 trillion parameters and weighs about 560 gigabytes. The company's approach tackles one of the biggest headaches in AI deployment: these enormous models require expensive hardware to run, and traditional setups waste resources by treating all processing stages the same way.

What Makes Running Trillion-Parameter Models So Expensive?

Large language models (LLMs) are neural networks trained on vast amounts of text data to generate human-like responses. Models like Kimi K2.5 are so massive that they cannot fit on a single graphics processing unit (GPU), the specialized chips that power AI inference. Instead, they must be split across multiple GPUs, requiring at least eight H100 GPUs (among the most expensive chips available) just to load the model into memory before any actual processing begins.

The challenge intensifies because different stages of processing have different computational needs. When a model processes your input text, it performs heavy mathematical calculations. When it generates output, it becomes memory-bound, meaning the bottleneck shifts from computing power to how fast data can move through the system. Traditional setups treat both stages identically, leaving expensive hardware idle or underutilized.

How Is Cloudflare Solving the Infrastructure Problem?

Cloudflare introduced three key innovations to address these inefficiencies:

• Disaggregated Prefill: The company separates model processing into two stages handled by different machines. One system reads and prepares input tokens while populating the key-value (KV) cache, a temporary memory structure that stores information needed for output generation. The other system generates the actual output tokens. This allows each stage to use hardware optimized for its specific computational demands.
• Infire Custom Inference Engine: Cloudflare built a proprietary engine that runs large language models across multiple GPUs more efficiently. Infire uses pipeline parallelism to prevent some GPUs from sitting idle while others work, and tensor parallelism to minimize the communication overhead between GPUs. The result is faster responses and reduced memory consumption.
• Unweight Compression System: A new compression technique that reduces model weights by 15 to 22 percent without sacrificing accuracy, allowing GPUs to load and move less data during inference.

"For pipeline parallelism, Infire attempts to properly load balance all stages of the pipeline, in order to prevent the GPUs of one stage from starving while other stages are executing. On the other hand, for tensor parallelism, Infire optimizes for reducing cross-GPU communication, making it as fast as possible," explained Michelle Chen, principal product manager at Cloudflare, alongside Kevin Flansburg, senior engineering manager, and Vlad Krasnov, principal systems engineer.

Michelle Chen, Principal Product Manager at Cloudflare

What Real-World Impact Do These Optimizations Deliver?

The practical results are striking. Cloudflare reports that Infire can now run Kimi K2.5 on just eight H100 GPUs while still leaving memory available for the KV cache, compared to the baseline requirement of eight H100s just to load the model. The company also demonstrated that it can run Llama 4 Scout, another large model, on only two H200 GPUs (which have larger memory capacity) with substantial room for context tokens.

These optimizations matter because they directly translate to lower operational costs for companies deploying AI services. Fewer GPUs required means less electricity consumed, less cooling infrastructure needed, and lower capital expenditure on hardware. For startups and enterprises building AI applications, this efficiency gain could mean the difference between a profitable service and one that bleeds money.

Steps to Optimize Large Language Model Deployment

Organizations looking to run massive AI models more efficiently can apply these principles:

• Separate Processing Stages: Identify which parts of your inference pipeline are compute-bound versus memory-bound, then allocate hardware resources accordingly rather than using uniform configurations across all stages.
• Implement Custom Inference Engines: Consider building or adopting inference engines that support both pipeline parallelism and tensor parallelism to maximize GPU utilization and minimize cross-GPU communication delays.
• Apply Model Compression Techniques: Explore weight compression and quantization methods that reduce model size by 15 to 22 percent without degrading output quality, allowing faster data movement through your system.

Why Is This Timing Critical for the AI Industry?

The infrastructure challenges Cloudflare is addressing reflect a broader industry problem. According to Cockroach Labs' recent State of AI Infrastructure report, many companies are discovering that their existing systems were not designed to handle the scale and unpredictability of AI workloads. Legacy infrastructure built around episodic human interaction simply cannot sustain the constant, high-volume demands of production AI systems.

Cloudflare's innovations suggest that the bottleneck in AI deployment is shifting from raw model capability to infrastructure efficiency. As models like Kimi K2.5 become more powerful and more companies want to deploy them, the ability to run these systems cost-effectively becomes a competitive advantage. Companies that can serve AI models with lower latency and lower operational costs will have more room to invest in other areas or pass savings to customers.

The work also highlights why infrastructure providers are becoming increasingly important players in the AI race. While model developers like Moonshot AI focus on training better algorithms, companies like Cloudflare are solving the unglamorous but critical problem of making those models practical to deploy at scale. Both pieces are essential for AI to move from research labs into everyday applications.