The real bottleneck for running large language models locally has shifted from computing speed to available memory. As open-weight AI models expand to 200 billion, 600 billion, or more parameters, whether a model fits on your machine at all now matters far more than how quickly it processes tokens. This fundamental change is reshaping how people choose hardware for on-device AI work, from casual enthusiasts downloading models on Friday nights to developers building production systems.

Why Memory Capacity Trumps Raw Computing Power for Local LLMs?

The shift reflects a dramatic expansion in model sizes available to run locally. Just a few years ago, the conversation centered on whether an 8 billion or 14 billion parameter model could fit on a consumer graphics processing unit (GPU). Today, open-weight models span vastly different architectures and scales. OpenAI's gpt-oss-20b occupies just 12.8 gigabytes, while gpt-oss-120b requires 60.8 gigabytes despite containing 116.8 billion parameters because most of its mixture-of-experts weights use a compressed format called MXFP4.

Move further up the scale and the memory demands become dramatic. Qwen3-235B-A22B contains 235 billion parameters with 22 billion active at any time, while DeepSeek-R1 reaches 671 billion total parameters and activates 37 billion. At 4-bit quantization, a compressed format that reduces model size, DeepSeek-R1 requires roughly 336 gigabytes before accounting for overhead like context length and temporary computation buffers.

This is why memory capacity now determines which models are available to you in the first place. Speed only becomes useful after a model actually fits into your hardware's memory pool.

How to Evaluate Local AI Hardware for Your Needs?

• Assess Your Model Size Target: Decide which models you want to run, then check their memory requirements. A 70 billion parameter model in 8-bit format needs roughly 70 gigabytes, while a 235 billion parameter model at 4-bit quantization requires around 118 gigabytes before overhead.
• Prioritize Memory Bandwidth: Memory bandwidth, measured in gigabytes per second, determines how fast data moves between storage and processing cores. Higher bandwidth becomes critical during token-by-token generation, where the system produces one word at a time and must repeatedly access model weights.
• Consider Your Workflow Type: Single-user inference on large models benefits most from maximum memory capacity and bandwidth. Development work involving fine-tuning or specialized quantization formats may benefit from access to CUDA, NVIDIA's software ecosystem that most AI research tools support natively.

Apple's new Mac Studio M5 Ultra exemplifies the memory-first approach. The machine offers up to 512 gigabytes of unified memory and 1.2 terabytes per second of memory bandwidth, positioning it around running enormous models locally on a single machine. NVIDIA's DGX Spark takes the opposite approach, offering 128 gigabytes of unified memory and 273 gigabytes per second of bandwidth, but pairing it with Blackwell compute architecture and CUDA support designed for development workflows.

A single DGX Spark is a capable 128-gigabyte AI machine. A 256-gigabyte or 512-gigabyte M5 Ultra belongs to an entirely different memory class. That distinction matters more now because open-weight models are spreading across very different architectures and parameter ranges.

What Do the Benchmarks Actually Show About Real-World Performance?

When researchers at LMSYS loaded Llama 3.1 70B in 8-bit format onto the DGX Spark, they measured 803 tokens per second during the prefill stage, which is when the system processes your prompt, documents, conversation history, and system instructions before producing the first token. Prefill benefits heavily from raw compute power, and Blackwell Tensor Cores excel at this work.

Decode, what happens afterwards as the answer arrives token by token, tells a different story. Large-model inference at low batch sizes spends most of its time moving weights through memory rather than performing calculations. Apple's 1.2 terabyte per second memory system offers roughly 4.4 times as much memory bandwidth as Spark's 273 gigabytes per second. Apple also offers up to four times Spark's memory capacity on the 512-gigabyte configuration.

Apple's own M5 Ultra announcement specifically highlights higher tokens-per-second performance from the 1.2 terabyte per second memory system and the ability to run large language models with hundreds of billions of parameters entirely on-device. However, important caveats apply. The M5 Ultra Mac Studio does not reach customers until September 22, 2026, with the 512-gigabyte version arriving in late October. Apple claims large language model prompt processing in LM Studio, a popular local AI application, is up to four times faster than on M3 Ultra, but that is a prefill result specific to prompt processing. Translating that into a predicted fourfold improvement in generation speed would be incorrect. The 1.2 terabyte per second figure is real, but what software manages to extract from it still requires testing.

NVIDIA's DGX Spark makes much more sense when you are building AI systems rather than simply running them. Open PyTorch, TensorRT-LLM, vLLM, or SGLang, and the argument changes immediately. Spark gives developers a compact Blackwell system running NVIDIA's CUDA ecosystem, with native low-precision support and tools that resemble what they will eventually use on larger NVIDIA hardware. NVIDIA already publishes Spark-specific recipes for supervised fine-tuning, LoRA, and QLoRA, including Llama 3.1 70B.

MLX, llama.cpp, Ollama, and LM Studio have made Apple Silicon dramatically more useful for local AI, and the M5 Ultra's memory architecture gives those tools remarkable hardware to work with. New AI research and production tooling still tends to arrive in CUDA first, especially when it involves new quantization formats, fused kernels, or specialized mixture-of-experts optimizations.

For someone developing a model that will later run on NVIDIA infrastructure, Spark is a small version of the environment they actually care about. For someone downloading a 200 billion parameter model on Friday night because they want to run it privately on one desk, that advantage matters far less.

NVIDIA does offer Spark buyers a clever upgrade path. Each unit carries a 200 gigabit per second network interface, and two machines can be connected for distributed workloads. NVIDIA says a pair can work with models up to around 405 billion parameters. At roughly $9,400 for two machines, that puts a dual-Spark setup into roughly the same spending territory as a high-memory Mac Studio. However, two Spark systems still have separate 128-gigabyte memory pools connected over a network. Their 273 gigabyte per second memory bandwidth does not combine into one 546 gigabyte per second unified bus, and distributing a model introduces communication overhead.

The practical implication is clear: as open-weight models continue to grow in size and capability, the machine that can hold the largest model in a single unified memory pool gains a decisive advantage for local inference work. Speed matters, but only after the model fits.