Mistral AI's Mixtral 8x22B is a powerful open-weight model, but its memory demands don't match what the parameter count suggests. The model has 141 billion total parameters, yet only 39 billion activate per token. That efficiency trick saves compute costs, but it creates a critical misconception: the active parameter count does not determine how much GPU memory you need.

At FP8 precision, Mixtral 8x22B requires roughly 162 gigabytes of video RAM (VRAM) before accounting for the key-value cache that stores context during generation. At BF16 precision, the requirement jumps to approximately 280 gigabytes. Neither configuration fits on a single GPU, and both demand multiple high-end accelerators to deploy.

Why Does Active Parameter Count Mislead?

The confusion stems from how mixture-of-experts (MoE) models work. Mixtral 8x22B contains eight distinct expert networks in each feedforward layer. A router network selects the top two experts for every token, and only those selected experts perform computation on that token. This sparse activation is why the model is called "8x22B" and why it has roughly 39 billion active parameters per forward pass.

However, the router cannot predict in advance which experts a given request will need. Across a full generation of any meaningful length, it is nearly certain that all eight experts will be used somewhere in the sequence. Because there is no way to know which experts will activate, all 141 billion parameters must remain in GPU memory before generation starts. The 39 billion active count only tells you how many parameters participate in producing any single token, not how much memory the model occupies.

This distinction explains why community discussions around Mixtral 8x22B consistently include people attempting to run it on a single 40GB or 48GB GPU and hitting an out-of-memory error immediately, sometimes after already seeing the model report a deceptively small active-parameter figure.

What GPU Setup Actually Works?

Deploying Mixtral 8x22B requires careful hardware planning. At FP8 precision, the model fits on four NVIDIA A100 80GB GPUs combined, which offer 320 gigabytes of total memory. At BF16 precision, four A100s become uncomfortably tight once real traffic hits the server; five A100 80GB GPUs, providing 400 gigabytes of memory, give the configuration actual working headroom for key-value cache and concurrent requests.

One important caveat: A100 GPUs lack native FP8 Tensor Cores, so FP8 inference on A100 runs through software emulation rather than hardware acceleration. It still fits the memory budget and still works, but it delivers lower throughput than the same FP8 configuration would achieve on H100 or newer hardware. If raw BF16 throughput matters more than cost, that is the tradeoff to weigh against A100's lower hourly price.

How to Deploy Mixtral 8x22B Across Multiple GPUs

• Enable Expert Parallelism: Most Mixtral tutorials skip a critical detail: tensor parallelism alone is not the right configuration for a sparse MoE model. vLLM supports expert parallelism specifically for MoE architectures, and combining it with tensor parallelism is what actually makes a multi-GPU Mixtral deployment work well. Without expert parallelism enabled, vLLM tensor-parallelizes every layer uniformly, including the expert layers, which is workable but inefficient.
• Choose Your Precision Carefully: FP8 precision reduces memory footprint to 162GB, fitting on four A100 80GB GPUs, but requires accepting lower throughput on A100 hardware. BF16 precision delivers better throughput but requires 280GB of memory, necessitating five A100 80GB GPUs for practical deployment with headroom for real traffic.
• Plan for Key-Value Cache: The 162GB and 280GB figures represent weight storage plus standard framework overhead for CUDA kernels, activation buffers, and allocator fragmentation. These calculations do not include key-value cache, which grows with context length and concurrent requests. Budget additional memory accordingly when planning your deployment.

The vLLM command that enables expert parallelism looks like this: include the flag "--enable-expert-parallel" alongside "--tensor-parallel-size 4" and your chosen precision. With expert parallelism enabled, individual experts can be distributed across GPUs more directly, which is a better fit for how the model's sparsity actually works.

What Happened to Mixtral 8x7B?

Mixtral 8x7B, the model that first brought sparse MoE to open-weight large language models (LLMs) in December 2023, was officially retired by Mistral in March 2025. Mistral's own documentation points to Mistral Small 4 as the replacement. If you are comparing Mistral and Mixtral options today, 8x7B is a historically important model, not a current self-hosting recommendation.

Mixtral 8x22B remains the MoE model Mistral still actively supports. Released in April 2024 under the Apache 2.0 open-source license, it handles English, French, German, Italian, and Spanish, includes native function calling, and supports a 64,000-token context window, roughly equivalent to processing 100,000 words at once. On benchmarks at the time of release, it outperformed the earlier Mixtral 8x7B and approached GPT-4-class performance on several standard evaluations.

Why Does Mixture of Experts Matter Beyond Mixtral?

The active-versus-total parameter distinction is not specific to Mixtral; it applies to every sparse MoE model. DeepSeek-R1, for example, activates only about 37 billion of its 671 billion parameters per token, a design that delivers the knowledge of a massive model with the running cost of a much smaller one. This decoupling between model size and compute cost is the core reason the MoE architecture has spread so quickly across the field.

Sparse activation lets a small active footprint deliver the quality once reserved for much larger dense models. The active-to-total ratio keeps dropping, from around 25 percent in earlier Mixtral designs to roughly 3 percent in newer designs. A lower ratio means more stored knowledge per unit of compute spent at inference. The tradeoff is that all those experts still occupy memory even when idle. This tension between compute savings and memory cost runs through every MoE deployment decision.

For teams considering open-weight MoE models, the key takeaway is straightforward: do not confuse active parameters with memory requirements. Plan your hardware based on total parameters, not active ones, and budget additional memory for key-value cache and framework overhead. With proper configuration and the right GPU setup, Mixtral 8x22B delivers capable inference at a meaningful compute cost advantage over dense models of comparable quality.