Apple's M5 Max and M5 Pro chips are proving far more capable at running large language models locally than previously thought, thanks to specialized software that treats optimization as a hardware problem. A new inference engine called Splash, built by Inco AI, is reporting speeds of 74 tokens per second on an M5 Pro with 48GB of unified memory when running Qwen3.8, a 27-billion-parameter model. That is nearly double the speed of existing general-purpose inference engines on the same hardware.

The breakthrough reveals something important about the future of local AI on consumer hardware: raw chip performance matters less than software design choices. Splash achieves its speed advantage not through hidden Mac features or new GPU capabilities, but by abandoning the flexibility that makes general-purpose inference engines useful. Instead of trying to support thousands of models across multiple quantization formats, Splash optimizes itself around one specific model at a time.

Why Does Splash Run So Much Faster Than Other Engines?

The answer lies in specialization. Most local inference engines like llama.cpp and Ollama are designed to be flexible. They support many different model architectures, quantization formats, context lengths, and hardware platforms. That flexibility comes with a cost: generic code paths, kernels that handle multiple tensor shapes, and memory planners that work across different models.

Splash takes the opposite approach. For each supported model, Inco specializes almost everything: the kernels, the draft model, the memory plan, and the performance baselines. The Splash package for Qwen3.8 is about 17.4 gigabytes and includes not just the model weights but also a dedicated speculative draft model, the vision encoder, and the tokenizer, all tuned to work together.

The most significant optimization is speculative decoding, which Splash treats as the default decode path rather than an optional feature. Instead of generating one token at a time, a smaller draft model predicts several likely future tokens, and the main model verifies those candidates together. When several guesses are correct, the system effectively moves forward by multiple tokens in one verification step. Because Splash's draft model is trained specifically for Qwen3.8, it predicts more accurately than generic drafts, saving more time than it costs.

The kernels themselves are shape-specialized. Inco generates and tunes Metal kernels for the exact dimensions used by Qwen3.8. Prefill operations get kernels designed for large batches, decode gets kernels designed for small verification blocks, and hybrid recurrent components get their own specialized paths. When the runtime knows the tensor dimensions ahead of time, it has fewer compromises to make.

What Do These Performance Numbers Mean in Practice?

The headline benchmark is 74 tokens per second on short prompts with Qwen3.8 on an M5 Pro. At 32,000-token context length, Splash holds around 54 tokens per second, while the next-fastest comparison engine achieved roughly 28 tokens per second. When four concurrent requests run together, Splash reaches about 170 tokens per second aggregate throughput, compared to roughly 43 tokens per second for the next-fastest engine.

These numbers come from Inco's own benchmarks, not independent testing, so they should be interpreted with that caveat in mind. The tests used specific coding prompts, recommended settings for each engine, reasoning enabled, and Inco's own Splash package. However, even accounting for favorable testing conditions, the architecture behind the results suggests a meaningful shift in how local AI inference could work.

The 48GB unified memory recommendation is not because Qwen3.8 itself needs that much space. The model package is only 17.4 gigabytes. Splash requires at least 36GB of unified memory because running the model is only part of the memory budget. The system also needs room for the target weights, the draft model, key-value cache for attention, recurrent state, batching, and application overhead.

How to Understand the Shift Toward Specialized Inference Engines

• Flexibility vs. Performance Trade-off: General-purpose engines like llama.cpp support thousands of models but pay a performance penalty. Specialized engines like Splash optimize for one model and achieve significantly higher throughput, suggesting a future where different inference engines target different popular models.
• Model-Specific Packaging: Splash packages include not just weights but also draft models, vision encoders, and tokenizers tuned for that specific target. This is fundamentally different from loading a standard model checkpoint into any inference engine, indicating a shift toward integrated, optimized packages.
• Software as Hardware Optimization: The speed gains come entirely from software design, not new chip features. This suggests that Apple Silicon's unified memory architecture is underutilized by general-purpose engines, and specialized software can unlock significant performance without hardware changes.
• Speculative Decoding as Default: Rather than treating speculative decoding as an optional feature, Splash makes it the normal decode path with a draft model trained for the target. This architectural choice, combined with shape-specialized kernels, compounds the performance advantage.

The broader implication is that the local AI inference landscape may fragment. Instead of one or two dominant general-purpose engines, we might see a future where popular models like Qwen3.8, Llama, and others each have specialized inference engines optimized specifically for them. For users with M5 Pro or M5 Max Macs, this could mean dramatically faster local inference without upgrading hardware.

Splash is currently integrated into LM Studio, a popular local AI interface, as a first-class runtime option. Users can download Splash, load a supported Qwen package, and access the performance gains through a simple button in the interface. The experience looks like another runtime option, but underneath is a fundamentally different inference philosophy.

What About Other Quantization Approaches on Apple Silicon?

While Splash focuses on optimizing inference engines, other researchers are pursuing aggressive quantization to shrink models for Apple Silicon. PrismML released Ternary Bonsai 2 27B on September 17, 2026, a model compressed to just 5.9 gigabytes using 1.76-bit ternary quantization. On an M5 Max, it delivers 46.8 tokens per second while maintaining 98.2% of the performance of the full-precision baseline.

Ternary quantization compresses weights to just three possible values: negative one, zero, or positive one. This is far more aggressive than the four-bit quantization that became the industry standard around 2024. The challenge with sub-four-bit quantization is that certain activation channels develop magnitudes orders of magnitude larger than their neighbors, causing catastrophic precision collapse when quantized uniformly.

PrismML solved this by applying rotation-based quantization, using Walsh-Hadamard matrices to spread sparse activation spikes uniformly across all hidden dimensions before projecting weights to ternary values. The result is that Ternary Bonsai 2 scores 83.9 on PrismML's 20-benchmark suite versus 85.4 for the uncompressed baseline, a gap of just 1.5 percentage points.

For users with M5 Max or M5 Pro Macs, this means a choice between two optimization strategies: run a full-size model with specialized inference software like Splash for maximum speed, or run a heavily quantized model like Ternary Bonsai 2 for maximum portability and lower memory requirements. Both approaches are viable on Apple Silicon, and both are improving rapidly.

The convergence of specialized inference engines and aggressive quantization suggests that Apple Silicon is becoming a serious platform for local large language model inference. Neither approach requires new hardware; both rely on better software and model compression techniques. For developers and users building local AI applications, Apple's unified memory architecture is proving to be a significant advantage.