Perplexity has demonstrated that the software layer controlling how AI models use computer chips matters just as much as the hardware itself. The company built a custom inference engine called Lily that runs a 35-billion-parameter model on Apple's M5 Max chip up to 1.32 times faster than existing general-purpose frameworks, without any hardware upgrades. This breakthrough reveals that as AI companies race to move powerful language models from data centers to consumer devices, the real innovation may lie in rewriting the software that sits between the model and the chip.

What Is Inference and Why Does It Matter?

When people talk about running an AI model on a device, they're usually referring to inference, which is the process of using a trained model to generate responses. This differs from training, the computationally expensive process of teaching a model using massive datasets in data centers. Inference happens every time you type a prompt and wait for an answer. Unlike training, inference can happen on edge devices like laptops and phones, as long as the required models fit in memory.

Inference itself has two distinct stages that behave differently on hardware. The prefill stage occurs when the model reads and processes everything you've typed. The decode stage follows, when the model generates your response one word at a time. Prefill can be processed in parallel because the model has the entire prompt available upfront. Decode is inherently slower because each new word depends on previously generated words, forcing the model to repeatedly access memory for information.

How Did Perplexity Achieve These Speed Gains?

Perplexity's engineers built Lily specifically for one model, China's Qwen3.6-35B-A3B, running on Apple's chips. On a Mac with an M5 Max chip, Lily processed prompts up to 1.23 times faster and generated responses up to 1.32 times faster than MLX-LM, a general-purpose framework built by Apple for running AI models on its hardware, according to Perplexity's testing across different prompt lengths.

The Qwen model presented unique optimization challenges. It uses a mixture-of-experts architecture, meaning it activates only a small, relevant subset of its 35 billion parameters for each word generated, rather than using all of them. This makes the model cheaper to run than its size suggests, but it creates unpredictable computing patterns that general-purpose engines struggle to optimize. The model also uses two different types of internal layers: some employ full attention, which reviews everything said so far in a conversation, while others use Gated DeltaNet, a newer method that keeps a running summary instead of storing everything, making it faster and lighter on memory.

Because Lily was built for this specific model, Perplexity's engineers could make optimization decisions that general-purpose tools cannot. Two techniques stood out:

• Smart Decompression: Perplexity compressed the model from about 70 gigabytes to roughly 19.4 gigabytes using 4-bit quantization, a technique that shrinks a model similar to how a large photograph can be compressed into a smaller file. The innovation came in how the software decompresses data. Instead of decompressing each piece and writing the expanded numbers back to memory before calculations, Lily decompresses data inside the chip itself at the moment it's needed. This change alone made prompt processing up to 77.4 percent faster in one test.
• Memory Movement Optimization: Apple's chips use unified memory, where the central processing unit (CPU) and graphics processing unit (GPU) share the same memory pool instead of having separate ones. Perplexity reduced how often the model needed to move data in and out of memory during response generation by keeping more calculations on the GPU rather than sending data back to the CPU unnecessarily.

None of these improvements required changing the underlying M5 Max chip itself. What changed was how completely the software used the existing hardware.

Is This Approach Unique to Perplexity?

Perplexity's strategy reflects a broader industry pattern. US-based AI startup PrismML managed to run a compressed version of a 27-billion-parameter AI model locally on consumer devices without relying on cloud infrastructure. Apple's own machine-learning research team published similar work in 2024, taking Llama 3.1, an open-source model with 8 billion parameters, and running it on an older M1 Max chip at about 33 tokens of output per second using 4-bit quantization and a technique called stateful cache that reuses previously calculated information instead of recalculating it at every step.

Microsoft has taken a different approach with Phi Silica, a small AI model built into Windows Copilot+ PCs. Because these laptops include a dedicated neural processing unit (NPU), which is more limited than a full graphics chip but far more power-efficient, Microsoft designed Phi Silica specifically around that constraint. It uses speculative decoding, where a smaller, faster model guesses the next few words and a larger model quickly checks whether those guesses are correct, rather than having the larger model generate every word from scratch. This lets the NPU produce responses faster without needing to be more powerful.

How to Optimize AI Models for Consumer Hardware

• Build Custom Inference Engines: Creating software specifically designed for a particular model and hardware combination, rather than using general-purpose frameworks, allows engineers to make optimization decisions that unlock significantly better performance without hardware upgrades.
• Apply Model-Specific Compression: Use quantization techniques to shrink models to fit in consumer device memory, then optimize the decompression process to happen inside the chip itself rather than writing expanded data back to memory first.
• Minimize Memory Movement: Reduce unnecessary data transfers between CPU and GPU by keeping more calculations on the GPU, taking advantage of unified memory architectures where available.
• Match Architecture to Hardware Constraints: Design models around the specific limitations of target hardware, such as neural processing units in laptops, using techniques like speculative decoding to maximize efficiency.

The pattern emerging across Perplexity, Apple, Microsoft, and PrismML demonstrates a fundamental principle: the software layer sitting on top of a chip, deciding how a specific model's calculations map onto that specific hardware, determines how much of the chip's potential actually gets used. As AI companies continue pushing powerful models onto consumer devices, this software-hardware co-optimization may prove more important than raw processing power.

" }