A new open-source framework called OpenJarvis is making it practical to run personal AI agents directly on your device, without sending data to cloud servers. Developed by researchers at Stanford University's Hazy Research lab, OpenJarvis addresses a critical gap in the software stack needed to make local-first AI actually work in the real world. The framework is motivated by a key finding: local AI models already handle most of the queries people actually ask them to do.

Why Should You Care About AI Running Locally on Your Device?

The shift toward on-device AI inference represents a fundamental change in how we interact with artificial intelligence. Instead of your laptop or phone sending every request to a distant data center and waiting for a response, the AI model runs right there on your hardware. This matters because it changes three critical things: speed, privacy, and reliability. Your AI assistant responds instantly without network latency, your personal data never leaves your device, and you can use AI features even when your internet connection is spotty or nonexistent.

For context, consider how AI hardware accelerators work. Devices equipped with Neural Processing Units, or NPUs, are specialized chips designed specifically for running AI workloads locally. An NPU's performance is measured in TOPS, which stands for "trillions of operations per second." For most consumer AI tasks, 40 TOPS is sufficient, and it's the minimum required for a device to earn Copilot+ certification from Microsoft. The key advantage of NPUs over general-purpose processors is their efficiency; they consume far less power while delivering the same AI performance, which means your battery lasts all day instead of draining in hours.

What Makes OpenJarvis Different From Other AI Frameworks?

OpenJarvis introduces three core design principles that set it apart. First, it emphasizes shared primitives, meaning developers can build on common building blocks rather than reinventing the wheel. Second, it takes an efficiency-first approach to evaluation, prioritizing how well the AI performs relative to the energy it consumes. Third, it enables continual self-improvement from local trace data, allowing the AI agent to learn and improve over time using information gathered directly on your device.

The framework also supports multiple inference backends, meaning it can work with different types of hardware and software configurations. This flexibility is crucial because not every device has the same processor or memory setup. Additionally, OpenJarvis includes an energy leaderboard, a public ranking system that tracks which models and configurations deliver the best performance per watt of power consumed.

How to Evaluate On-Device AI Performance for Your Needs

• Measure Response Speed: On-device inference typically delivers responses in milliseconds because data doesn't need to travel to a distant server and back. Compare this to cloud-based AI, which adds network latency and can feel sluggish on slower connections.
• Assess Privacy Requirements: If you're working with sensitive information, on-device processing keeps that data local. Your medical records, financial documents, or personal conversations never leave your device, which is especially important for regulated industries.
• Consider Battery Impact: NPU-based on-device AI consumes significantly less power than GPU-based alternatives. If all-day battery life matters to you, on-device inference is the better choice than cloud processing or local GPU computation.
• Check Offline Capability: On-device models work without an internet connection, making them valuable in areas with unreliable connectivity or when you need AI features in remote locations.

The distinction between on-device inference and cloud-based AI is important to understand. Inference is the process of using a trained AI model to produce results, as opposed to training, which is the process of teaching the model by feeding it data and adjusting its parameters. Most consumer devices focus on inference because training requires massive computational resources found only in data centers.

OpenJarvis is being presented at AMD's Advancing AI event in July 2026 by Jon Saad-Falcon and Avanika Narayan, both PhD candidates at Stanford's Scaling Intelligence Lab. The framework represents a shift in how the AI community thinks about where computation should happen. Rather than assuming all AI work belongs in the cloud, researchers are now asking: what can we do locally, and how do we make it efficient enough to be practical ?

The broader context matters here. The AI hardware accelerator landscape now includes three main types of specialized processors. NPUs handle on-device inference in consumer devices like laptops and phones. TPUs, or Tensor Processing Units, are Google's specialized chips designed for large-scale machine learning in data centers. GPUs, or Graphics Processing Units, offer versatile parallel computing power for both training and inference but consume significantly more energy than NPUs.

For everyday users, the practical implication is clear: your next laptop or phone will likely include an NPU, and frameworks like OpenJarvis will make it easier for developers to build AI features that run locally. This means faster, more private, and more reliable AI experiences without the constant need to send data to the cloud. The research finding that local models already handle most real-world queries suggests this shift isn't just technically possible; it's already overdue.