The bottleneck in AI infrastructure isn't building bigger clusters anymore; it's figuring out how to let multiple teams safely share the same expensive GPUs without stepping on each other's work. As AWS and NVIDIA announce plans to deploy 2 million additional graphics processors across AWS's global infrastructure by 2027-2028, the industry is quietly solving a different problem: how to turn a pile of accelerators into an "AI factory" that dozens of teams can draw from simultaneously.

An AI factory isn't just a model or a cluster. It's a shared pool of GPUs where one team might be fine-tuning a model, another serving inference requests to customers, and a third running evaluations, all on the same hardware at the same time. The hard part isn't training anymore. It's giving every team safe, isolated access to expensive accelerators without anyone accidentally consuming resources meant for someone else.

Why GPU Utilization Is the Real Metric That Matters?

Cloud providers and enterprises grade GPU infrastructure the same way: not by peak performance in a single run, but by utilization. SemiAnalysis's ClusterMAX scoring system rewards providers on security, networking, storage, reliability, and support rather than raw throughput. The wrapper around the GPUs, not the GPUs themselves, is what gets judged.

Two problems keep utilization low. First, the traditional resource model treats a GPU like an indivisible unit. If a pod requests one GPU, it pins the entire accelerator even if it only uses 10 percent of its capacity. Kubernetes 1.34 introduced Dynamic Resource Allocation (DRA), which lets the scheduler treat accelerators as rich devices with attributes, memory, and topology information, enabling better packing.

Second, isolation concerns push teams toward dedicated clusters or dedicated GPU blocks per team. This is the safe choice when trust boundaries are strict, but it wastes most of the hardware. The fix isn't a new model server. It's a stack that allocates accelerators so capacity is neither stranded nor unsafe, and isolates tenants so packing them together holds up.

What Building Blocks Make an AI Factory Work?

An AI factory is an assembly problem. Most layers are Kubernetes-native or Cloud Native Computing Foundation (CNCF) projects, with a few open-source tools filling gaps. The stack spans hardware lifecycle management, cluster orchestration, GPU allocation, inference serving, networking, storage, observability, and security.

Steps to Building an AI Factory on Kubernetes

• Hardware Provisioning: Discover each node's inventory (GPU count, memory health, network card identities), network boot the node with an OS image containing GPU drivers and CUDA libraries, apply BIOS settings for baseline or performance modes, run burn-in tests under load, and confirm GPUs communicate at full bandwidth using NCCL tests. Results are written to a source of truth like NetBox for IP address tracking.
• GPU Allocation and Scheduling: Use Dynamic Resource Allocation to treat accelerators as rich devices, deploy HAMi (a CNCF Incubating project) to enforce per-pod memory and compute limits in software so multiple pods run on one card with guardrails, and use schedulers like KAI Scheduler and Volcano for gang and topology-aware placement. Kueue handles queueing, admission, and quota management.
• Tenant Isolation: Deploy vCluster for tenant clusters or use sandboxed runtimes to keep teams apart on the same hardware. For confidential computing, place untrusted tenants on whole GPUs rather than partitioned ones. MIG (Multi-Instance GPU) isolates memory and faults in hardware but is typically reserved for workloads within a single trust domain.
• Inference and Serving: Run models behind an API using vLLM as a common inference engine, wrap it with KServe (a CNCF Incubating project) for autoscaling and standard endpoints, and use Gateway API with LiteLLM to add an OpenAI-compatible gateway so dozens of specialized models speak one API.
• Networking and Storage: Move data between GPUs using Cilium, Multus, and SR-IOV networking. Persist datasets, checkpoints, and models using Container Storage Interface (CSI), Rook/Ceph, or object storage. Accelerate data processing with NVIDIA cuDF and vector indexing with NVIDIA cuVS CUDA-X libraries.

How Are AWS and NVIDIA Scaling This Infrastructure?

AWS and NVIDIA's expanded collaboration reflects the urgency of this problem. The companies plan to deploy 2 million additional NVIDIA Blackwell Ultra, Rubin, and Rubin Ultra GPUs across AWS's global infrastructure in 2027-2028, including AI factories for the U.S. government with 100,000 GPUs on secure AWS infrastructure for federal and national-security workloads.

Beyond raw GPU count, the partnership extends to heterogeneous infrastructure. AWS will bring NVIDIA Vera CPU-based infrastructure to its platform, providing high-performance CPU compute alongside accelerators for agentic AI workloads. NVIDIA and Amazon's Annapurna Labs are expanding support for NVIDIA NVLink Fusion high-speed chip interconnect technology to work with NVIDIA's custom high-bandwidth memory (NVHBM), giving AWS Trainium chips access to faster, more power-efficient memory.

The companies are also collaborating on data processing and analytics. GPU-accelerated data processing on Amazon EMR using Amazon EC2 G7 instances and NVIDIA cuDF delivers up to 3.7 times faster processing speeds and 30 percent better price performance compared to CPU-based configurations. GPU-accelerated vector indexing on Amazon OpenSearch Service delivers up to 9 times faster vector indexing at a quarter of the cost.

Why Does This Matter for Enterprises and Governments?

The shift toward AI factories reflects a maturation in how organizations think about AI infrastructure. Two years ago, every platform team was building a developer platform from scratch. Kubernetes already had mature primitives for containers, role-based access control (RBAC), autoscaling, and policy, but it lacked a clean answer for accelerators and keeping tenants apart on the same nodes. That gap is now closing.

For enterprises, this means moving from pilot to production at scale. Customers are scaling workloads across agentic AI, scientific discovery, enterprise automation, and robotics. They need broader model choice, faster data pipelines, and confidence that underlying infrastructure can keep pace with innovation while maintaining security and reliability for mission-critical workloads.

For governments, the stakes are even higher. AWS and NVIDIA are building AI factories for the U.S. government capable of running workloads classified at Impact Level 6 (IL6) and above, putting the partnership at the center of federal AI advancement for national security.

"An AI factory is not just a model or a cluster. It is a pool of GPUs that many teams draw from at once: one team fine-tuning, another serving inference, a third running evaluations, all on the same accelerators," explained Hrittik Roy, CNCF Ambassador and Platform Advocate at vCluster.

Hrittik Roy, CNCF Ambassador and Platform Advocate at vCluster

The real economics of AI infrastructure come down to utilization. A single GPU sitting idle because it's reserved for one team is wasted capital. The cloud-native ecosystem now supplies most of the building blocks to solve this problem: Kubernetes orchestration, NVIDIA's GPU management tools, CNCF projects like KServe and HAMi, and open-source solutions like vCluster and Dynamo. The companies and teams that master this assembly will own the next generation of AI infrastructure.