Ollama has dominated local AI for its simplicity, but a growing number of developers are discovering that convenience comes at a real performance cost. One tech journalist recently switched from Ollama to BaseRT, a newer runtime built specifically for Apple Silicon, and found that model startup times dropped dramatically. The shift highlights a broader tension in the self-hosted AI space: ease of use versus raw speed.

Why Is Ollama Losing Ground to Specialized Runtimes?

Ollama has become the default choice for running large language models (LLMs) locally because it handles nearly everything automatically. Users can download a model with a single command, run it immediately, and connect it to other applications through a local API (Application Programming Interface) without worrying about model formats or hardware configuration. However, this convenience comes with hidden trade-offs.

The core issue is that Ollama unloads models from memory when they haven't been used for a while. When you want to use the model again, it needs to reload, which adds a noticeable delay before it can answer your question. For casual users, this might be acceptable. But for developers using AI coding agents that repeatedly send files, tool results, and conversation history back to the model, the delay becomes frustrating.

Ollama also needs to work across multiple operating systems and hardware platforms, including macOS, Windows, Linux, Apple Silicon, Nvidia GPUs, and many others. This broad compatibility means the runtime follows a general path to access hardware rather than optimizing for any single chip. That's a smart design choice for a universal tool, but it leaves performance on the table for users with specific hardware.

How Does BaseRT Achieve Faster Performance?

BaseRT takes a fundamentally different approach. It's written directly against Apple's Metal API and doesn't depend on intermediate frameworks like MLX, PyTorch, or Core ML. Instead, it uses custom kernels designed around Apple's unified memory architecture, which reduces computational overhead.

The performance gains are most dramatic during the "prefill" stage, when a model reads everything you send it before generating a response. This includes your question, conversation history, documents, code, and any instructions the application adds. How long you wait for the first token to appear depends heavily on prefill speed. BaseRT's published benchmarks on the M5 Pro show up to 6.4x faster prefill than llama.cpp and up to 3.9x faster prefill than MLX.

BaseRT achieves this by accessing the neural accelerators built into every GPU core on M5 chips through Metal 4. These accelerators handle the matrix calculations involved in prompt processing, and the advantage grows larger with longer prompts and mixture-of-experts models. For coding agents that frequently send large amounts of code and conversation history back to the model, this difference is noticeable in daily use.

Token generation speed improvements are smaller but still meaningful. BaseRT claims up to 1.75x faster generation than llama.cpp and up to 1.33x faster generation than MLX on an M5 Pro. While the difference may not feel dramatic once an answer is already streaming, getting to that first token sooner makes a real difference when you use AI tools throughout the day.

Steps to Get Started with BaseRT on Apple Silicon

• System Requirements: BaseRT requires an Apple Silicon Mac running macOS 14 or later. Installation is straightforward through the Terminal command line interface.
• Model Conversion: The first time you pull a model, BaseRT downloads it and converts it into its own.base format, which supports quantization levels from Q2 to Q8 and is optimized for how BaseRT loads and runs models.
• API Integration: BaseRT provides an OpenAI-compatible API, so any application that supports a custom OpenAI endpoint can connect to http://127.0.0.1:8080/v1, including chat interfaces, scripts, and coding tools.
• Model Management: You can view all installed models with the basert list command and convert existing GGUF, MLX, or Hugging Face checkpoints instead of downloading new copies.

What Are the Trade-Offs?

BaseRT is not a universal replacement for Ollama. It only works on Apple Silicon Macs, while Ollama runs on Windows, Linux, and Intel-based systems. Model compatibility also isn't as comprehensive as Ollama's yet, meaning some models that work with Ollama may not be available for BaseRT.

Ollama remains the better choice for users who prioritize simplicity and broad compatibility. It picks the model format, configures hardware acceleration, manages memory, and provides sensible defaults without requiring users to understand the underlying technical details. For most people, Ollama is still the right tool.

However, developers who want maximum performance and are willing to learn a specialized tool have alternatives. Beyond BaseRT, Docker Model Runner offers another option for those who already use Docker to self-host applications. It lets developers declare AI models directly inside Docker Compose alongside their application stack, making it convenient for developers with existing containerized workflows.

What Does This Mean for the Local AI Ecosystem?

The emergence of specialized runtimes like BaseRT suggests that the local AI market is maturing beyond one-size-fits-all solutions. As more developers move beyond simple chatbot use cases into production applications and coding agents, the demand for optimized, hardware-specific tools will likely grow. Ollama has introduced an MLX backend for supported models on Apple Silicon, which should close some of the performance gap, though it remains in preview and doesn't cover everything in Ollama's library.

The broader lesson is that convenience and performance are not always aligned. Ollama made local AI accessible to millions by handling complexity automatically. But as users become more sophisticated, they're discovering that those automatic choices sometimes sacrifice speed. For developers running coding agents or processing long documents repeatedly, that trade-off is no longer acceptable. The next generation of local AI tools will likely need to balance ease of use with the ability to unlock hardware-specific performance gains for users who need them.