Running large language models (LLMs) locally requires far more video memory (VRAM) than most guides claim, because the difference between "loading" a model and actually "running" it is the difference between a demo and a usable tool. A detailed breakdown of local AI hardware requirements reveals that the formula is simple arithmetic: model weights plus context cache plus operating system overhead must all fit in fast memory, or performance collapses.

Why "Runs on Your GPU" Claims Are Often Misleading?

The most common mistake in local AI guides is conflating "loads" with "runs." A model can technically load onto a graphics card through offloading, which moves weights between slow system RAM and the GPU as needed. But this approach creates a 3-10x slowdown compared to keeping everything in fast memory. For example, an RTX 4090 graphics card "running" a 70-billion-parameter model through offloading produces only about 1.6 tokens per second, meaning a short answer takes roughly 30 seconds. That is not a workflow anyone maintains past the first afternoon.

The honest test is straightforward: Is the entire model fully inside fast memory with room left for context? If not, the speed numbers quoted in listings reflect offloaded performance, and the system will feel broken during actual use.

What Is the Actual VRAM Formula for Local LLMs?

The calculation breaks down into three components. At the Q4_K_M quantization level (a compression format that balances quality and file size), budget approximately 0.6 gigabytes per billion parameters for model weights. Context cache, which stores the conversation history the model needs to reference, adds roughly 128-192 kilobytes per token at standard precision, meaning a 32,000-token context window adds 4-6 gigabytes on a mid-size model. Windows and desktop applications consume another 1-2 gigabytes.

The complete formula is: VRAM needed equals (parameters times quantization size) plus context cache plus operating system overhead. This arithmetic approach removes guesswork from hardware decisions.

How to Match Your Hardware to Real Model Performance

• 8GB VRAM: Runs 7B-8B models like Qwen 3 8B comfortably for chat, coding help, and agents. Produces roughly 30 tokens per second on an RTX 4060. Not sufficient for 12B models or long-context work.
• 12GB VRAM: Unlocks the 12B dense model class, which represents the biggest quality jump at the low end. Example: Gemma 4 12B produces approximately 42 tokens per second.
• 16GB VRAM: Adds 14B models and the first 20B-class mixture-of-experts (MoE) models, where sparse architectures activate only relevant parameters. Considered the value tier for enthusiasts.
• 24GB VRAM: The enthusiast sweet spot, running 27B dense models or fast 20B MoE variants at approximately 37 tokens per second. Example: Qwen 3.8 27B.
• 32GB VRAM: Reaches the 35B MoE tier, where sparse models begin to rival frontier model quality. Qwen 3.6 35B-A3B produces roughly 118 tokens per second.
• 48GB+ VRAM: Handles 70B models like Llama 3.3 70B at 10-23 tokens per second, depending on architecture.
• 96GB+ VRAM: Required for 120B MoE models like Qwen 3.5 122B-A10B, producing 27-41 tokens per second on a single consumer machine.

Apple Silicon unified memory maps directly onto this table. A 16-gigabyte Mac behaves like a 12-gigabyte GPU after macOS overhead. A 24-gigabyte Mac runs the same models as a 24-gigabyte GPU, though slower per byte but with identical capacity.

Why Context Length Changes Everything?

A critical oversight in most VRAM guides is ignoring context cache costs. A dense 27B model at 128,000-token context can consume 24 gigabytes or more of key-value cache at standard precision, which exceeds the model's own weights. A machine that technically "fits" the model can still fail under real workloads with long conversations or document processing.

However, newer hybrid linear-attention architectures like Qwen 3.5, Qwen 3.6, and Qwen 3-Next cache key-value data only on their sparse full-attention layers. This design reduces long-context memory cost by 4-13 times compared to dense models of the same size. For use cases involving long documents, this architectural difference matters as much as raw VRAM; in fact, it is often the deciding factor.

How to Evaluate Whether a Model Really Fits Your Hardware?

• Check the Quantization Level: A model at Q4 and the same model at Q2 are different products with different quality and file size. Guides must name the quantization to be credible.
• Distinguish Loading from Running: Ask whether every weight sits in fast memory with room left for context. This is testable by checking if the system maintains speed during actual conversations.
• Verify Speed Metrics: A fit verdict without tokens per second is only half the story. Speed determines whether the tool is usable or merely a demonstration.

If any of these three answers is missing from a claim, the math can be reproduced independently using the formula at the foundation of this analysis. The entire point of this breakdown is that VRAM decisions should rest on arithmetic, not faith or marketing claims.

For those choosing between platforms, a GPU wins for 8B-32B models at maximum speed due to superior memory bandwidth. For 70B-120B models, a Mac Studio is usually the cheaper path because it scales to much larger memory pools affordably.