A new framework called LiteRT-LM is making it practical for Android developers to run language models directly on phones, without sending data to the cloud or paying per-token fees. The tool handles the complex backend work of managing conversation memory and token streaming, leaving developers to focus on building responsive chat features that work offline and protect user privacy by design.

What Makes Running AI Locally on Your Phone So Hard?

Running a language model (LLM) on a mobile device sounds simple in theory but involves several technical hurdles that most developers aren't equipped to solve alone. The biggest challenge is managing the KV-cache, a type of memory that stores information about previous tokens to speed up generation. In a multi-turn conversation, this cache grows with each exchange, and developers must manually decide how much memory to allocate and when to clear it.

Beyond memory management, developers also need to handle token streaming, which means displaying the model's response word-by-word as it's generated rather than waiting for the entire answer to arrive. They must also choose whether to run the model on the phone's CPU or GPU, manage sampling parameters that control how creative or predictable the output is, and format system prompts and tool calls in the exact way their specific model was trained to understand them.

LiteRT-LM, Google's new framework, abstracts away much of this complexity. It wraps Google's existing LiteRT runtime with the orchestration logic that a chat application needs, handling KV-cache bookkeeping, session management, and token-by-token delivery automatically.

When Should You Choose On-Device AI Over Cloud APIs?

The decision to run AI locally versus in the cloud involves tradeoffs that vary by use case. On-device inference offers complete privacy, zero network latency, and no per-token billing, but it requires developers to own decisions about latency, memory usage, and sampling that cloud APIs would otherwise handle automatically.

Developers should reach for LiteRT-LM in these specific scenarios:

• Multi-turn conversations: When users need to maintain context across multiple exchanges, LiteRT-LM's session management handles KV-cache bookkeeping automatically, eliminating the need to replay the entire conversation history with each new message.
• Streaming output requirements: Applications that need token-by-token delivery for responsive chat interfaces benefit from LiteRT-LM's built-in streaming callbacks, which fire as soon as the model generates each token.
• Runtime backend flexibility: When a device's capabilities vary, LiteRT-LM lets developers choose between CPU and GPU execution at runtime rather than baking that decision into the app at build time.
• Pre-converted model availability: The LiteRT-LM Model Zoo includes ready-to-use versions of popular models like Gemma, Qwen, Llama, and Phi, eliminating the need for developers to perform their own PyTorch-to-LiteRT conversion work.
• Privacy-sensitive or latency-critical tasks: Requests that must never leave the device or that cannot tolerate network round-trip delays are ideal candidates for on-device inference.

However, on-device models are significantly less capable than the large models running on cloud servers. A developer might run a smaller Gemma variant locally for simple, fast tasks while falling back to a cloud LLM for requests that demand more reasoning power or knowledge.

How to Build a Chat App with LiteRT-LM

Implementing LiteRT-LM follows a straightforward three-step pattern, though developers need to understand the underlying architecture to avoid common pitfalls. The framework uses two key objects: an Engine that owns the loaded model and backend selection, and a Session that manages conversation state. The Engine is expensive to create and should be reused across conversations, while Sessions are cheap and disposable, meant to be created fresh for each new chat.

• Configure the Engine: Set the model path, choose between CPU or GPU backend, and define maxNumTokens as the combined budget for input and output tokens. For Gemma 4, a budget of 16,384 tokens worked in practice, though the right ceiling depends on the specific model and device memory constraints.
• Initialize and create a Session: Once the Engine is ready, create a Session with a SamplerConfig that defines how the model generates text. A solid default is topK of 64, topP of 0.95, and temperature of 1.0, though these values should be tuned per model family to avoid repetitive greedy output.
• Stream responses with callbacks: Call session.generateContentStream with your prompt and a ResponseCallback object. The onNext callback fires on an engine thread as each token arrives, so developers must hop to their UI dispatcher before updating the interface.

One critical gotcha: both the litert and litertlm-android libraries bundle a native library called libLiteRtClGlAccelerator.so, but the versions are not interchangeable. Using the wrong version can cause a SIGSEGV crash in nativeCreateEngine. The fix is a Gradle configuration that extracts the matching library from the litertlm-android archive into generated jniLibs, allowing source-set libraries to take precedence during the build.

System prompts and tool calling require manual assembly. Unlike cloud LLM APIs that accept structured system roles and function-calling schemas, LiteRT-LM takes a flat list of text inputs. Developers must format these inputs exactly as their model was trained to read them, using the correct turn-marker tokens and special characters. Gemma 1 through 3 use "" and "" markers, while Gemma 4 uses "<|turn>" and "" markers, so using the wrong prompt specification will degrade output quality.

What Are the Real-World Tradeoffs?

The decision to run AI on-device versus in the cloud mirrors the broader tradeoff between privacy and capability. Full on-device inference guarantees that nothing a user types ever leaves the device, and there are no per-token costs or network round-trips to wait for. But developers must own the latency, memory, and sampling decisions that a cloud LLM API would otherwise absorb.

An on-device model is nowhere near as capable as the giant models running behind a cloud API, but it's also free to run at inference time. This opens new possibilities for hybrid architectures where simple, latency-sensitive, or privacy-sensitive requests run locally, and more complex tasks fall back to the cloud when a local Gemma variant cannot deliver the required capability.

For developers building Android apps, LiteRT-LM removes the genuinely hard parts of on-device LLM inference, but it relocates rather than eliminates the work. Developers still need to size their token budget carefully, define sampling defaults, hand-build system prompts and tool calling, and watch for native library collisions at build time. Know those constraints going in, and the API itself becomes a clean, manageable pattern for building responsive, private AI features that work offline.