May 2026 brought a wave of meaningful upgrades across every major local AI runtime, with Ollama, vLLM, llama.cpp, MLX, and LM Studio all shipping real performance gains rather than incremental version bumps. For developers and organizations running AI models on their own hardware instead of relying on cloud services, these updates represent a significant shift in what's possible without leaving your infrastructure.

What Exactly Happened to Ollama in May?

Ollama, one of the most popular tools for running large language models locally, released six versions in just 11 days, signaling an unusually active development cycle. The standout achievement was Ollama 0.24.0, which introduced support for Codex App, OpenAI's desktop coding environment. This means developers can now run Codex against Ollama's open-source models, including Kimi-K2.6, GLM-5.1, Nemotron-3-Super, Gemma 4 31B, and Qwen 3.6, without manually configuring environment variables or custom endpoints.

But the real speed story came earlier in the month. Ollama 0.23.1 added Multi-Token Prediction (MTP) speculative decoding for Gemma 4 on Apple Silicon Macs, landing the same day Google released the necessary drafter weights. The result was striking: over 2x faster generation speeds on Gemma 4 31B coding tasks. The technique works by having a smaller "drafter" model predict multiple tokens ahead, which a larger "verifier" model then checks in a single pass, reusing cached computations to avoid redundant work.

Ollama also reworked its MLX sampler for improved generation quality on Apple Silicon and improved API response caching, achieving a median latency improvement of roughly 6.7x on model lookups. For developers using VS Code or other integrations, this means noticeably snappier performance on cold starts.

How Are Other Runtimes Keeping Pace?

Ollama wasn't alone in shipping meaningful improvements. vLLM, a runtime optimized for high-throughput serving on GPUs, released v0.21.0 on May 15, 2026, as a stabilization release focused on DeepSeek V4 performance. The update introduced a new TOKENSPEED_MLA attention backend for DeepSeek-R1 and Kimi-K2.5 models running on Blackwell GPUs, and fixed a correctness bug where speculative decoding wasn't properly respecting reasoning budgets on models designed to "think" before answering.

vLLM also integrated KV Offload with its Hybrid Memory Allocator, which improved throughput on models that were previously wasting significant KV-cache capacity. On certain architectures, this fix recovered up to 79.6% of wasted cache capacity. For context, vLLM benchmarks from May 2026 show roughly 2.3x higher throughput than Ollama under 8 concurrent users on Llama 3 8B, though the gap widens as concurrency increases.

llama.cpp, a lightweight C++ runtime, merged Multi-Token Prediction support for Qwen 3.6, achieving roughly 2x generation throughput in single-user scenarios on dense models. However, the update also revealed a practical limitation: on mixture-of-experts (MoE) models like Qwen 3.6 35B-A3B, the overhead of loading different expert slices for each drafted token can eliminate the speed gain entirely on consumer hardware running single-stream workloads. This trade-off highlights why local AI workflows often involve choosing between dense and MoE architectures.

MLX, Apple's machine learning framework, unlocked M5 Neural Accelerators in macOS 26.2, enabling up to 4x faster time-to-first-token (TTFT) on Apple's latest chips. LM Studio, a user-friendly desktop application, promoted Multi-Token Prediction speculative decoding to stable status in version 0.4.14 and added parallel vision predictions in 0.4.13.

What Do These Updates Mean for Self-Hosted AI?

The May 2026 update cycle reveals a maturing ecosystem where local AI runtimes are no longer just alternatives to cloud services, but genuinely competitive platforms with distinct performance profiles. The speed improvements, particularly the 2x gains on Apple Silicon and the stabilization of speculative decoding across multiple runtimes, suggest that running models locally is becoming faster and more practical for real-world workflows.

The addition of Codex App support to Ollama is particularly significant because it bridges the gap between open-source models and professional developer tools. Previously, developers wanting to use local models with Codex would need to manually configure endpoints and environment variables. Now, the integration is seamless, lowering the barrier to adoption for teams concerned about data privacy or cloud costs.

Steps to Evaluate Which Runtime Fits Your Needs

• Throughput Priority: If you're serving multiple concurrent users and need maximum tokens per second, vLLM's 2.3x throughput advantage over Ollama on Llama 3 8B makes it the stronger choice for production servers, though the gap widens significantly as concurrency increases.
• Apple Silicon Optimization: If you're running on a Mac with M-series chips, Ollama's MLX runner with Multi-Token Prediction support and MLX's M5 Neural Accelerator unlock offer the fastest local inference, with up to 4x speed improvements on newer hardware.
• Desktop Integration: If you want seamless integration with professional tools like Codex App or VS Code, Ollama's launch command ecosystem and improved API caching make it the most developer-friendly option for local workflows.
• Model Architecture Compatibility: If you're working with mixture-of-experts models on consumer hardware, be aware that Multi-Token Prediction may not provide speed gains; dense models like Qwen 3.6 27B see clearer benefits.
• Lightweight Deployment: If you need minimal dependencies and broad hardware support, llama.cpp's continuous build model and support for CUDA, Vulkan, HIP, and SYCL across Windows, macOS, and Linux offer maximum flexibility.

The broader pattern across all five runtimes is clear: speculative decoding, where a smaller model drafts tokens and a larger model verifies them, has moved from experimental to stable across the ecosystem. This technique is now the standard way to accelerate local inference, and May 2026 saw it mature significantly with bug fixes, better reasoning model support, and improved robustness to unusual chat templates and system prompts.

For organizations evaluating self-hosted AI infrastructure, the timing is noteworthy. The combination of faster hardware (M5 chips, Blackwell GPUs), more efficient runtimes, and better tooling integration suggests that the practical case for local AI is strengthening. The trade-off between cloud convenience and local control is shifting in favor of local deployment for teams with the infrastructure to support it.