NVIDIA has released two tools designed to make on-device AI faster and cheaper: Nemotron 3.5 Lightning, a 30-billion-parameter model optimized for local inference, and NeMo Switchyard, an open-source routing system that automatically directs each task to the most efficient model available. Together, they represent a shift in how enterprises are thinking about AI deployment, moving away from one-size-fits-all cloud solutions toward hybrid systems that balance cost, speed, and privacy.

Why Are Companies Moving AI Inference Back to Local Devices?

As artificial intelligence systems become more complex, organizations are increasingly deploying "always-on agents" that run continuously to handle specialized tasks like code review, security monitoring, and customer support. These systems don't need a single powerful model; instead, they work better as ensembles, where different models handle different jobs. A large reasoning model might orchestrate a workflow, while smaller, faster models handle the actual work. This architecture makes sense on local hardware because it reduces latency, keeps sensitive data private, and cuts cloud computing costs.

Nemotron 3.5 Lightning is built specifically for this use case. The model delivers up to 4 times faster output speed compared with competitors in its class, leading to 30% faster task completion for agentic workflows. Because it's open and customizable, organizations can fine-tune it on their own domain data, tools, and workflows to improve accuracy for specialized tasks without relying on external APIs.

How Does Intelligent Model Routing Save Money and Improve Speed?

NeMo Switchyard solves a practical problem: when you have access to multiple models of different sizes and capabilities, how do you decide which one to use for each request? Sending everything to your most powerful model wastes money. Sending everything to your cheapest model sacrifices quality. Switchyard automates this decision by routing each prompt to the most capable and efficient model for that specific task.

The results are striking. NVIDIA's internal benchmarks show that NeMo Switchyard maintains frontier-level accuracy while reducing task completion cost to nearly one-third of using Opus 4.8 alone, which represents approximately 67% savings. In real-world deployments, the savings vary by use case. LangChain achieved 74% lower cost in multi-turn agent tasks by routing only 7% of calls to a frontier model, accepting just a 6% accuracy tradeoff. Ramp matched a frontier model's performance while cutting costs by 58% and runtime by 33%.

Steps to Deploy Intelligent Routing in Your AI Workflows

• Evaluate your model ensemble: Identify which models you have access to, their capabilities, costs, and latency profiles. This might include open models like Nemotron 3.5 Lightning, proprietary models like GPT-5.6, and specialized models fine-tuned for your domain.
• Define routing priorities: Decide whether your primary concern is cost, latency, accuracy, or a combination. NeMo Switchyard lets you tune routing algorithms to match your specific priorities without rewriting your application code.
• Integrate with your existing stack: Switchyard works with popular agent frameworks like LangChain, LiteLLM, and Kong AI Gateway, so you can add intelligent routing without major architectural changes.
• Monitor and iterate: Use real-world performance data to refine your routing strategy. Partners like Boomi achieved 100% domain-routing accuracy and sent 59% of traffic to a faster fine-tuned model, reducing latency by 21%.

What Real-World Performance Gains Are Companies Seeing?

Early adopters across multiple industries are reporting significant improvements. CrowdStrike customized Nemotron 3.5 Lightning for cybersecurity tasks, while Harvey with Trajectory deployed it for legal services and CodeRabbit with Baseten optimized it for code review. Lila Sciences is using the model to improve reasoning capabilities for physical and life sciences workloads. Fastino Labs customized the model and achieved leading accuracies for software development, finance, and healthcare.

Beyond cost savings, companies are seeing speed improvements. Meta's Muse Glimmer, a 30-billion-parameter model optimized for coding and agentic AI, delivers over 200 tokens per second on an NVIDIA RTX 5090 GPU, making it practical for real-time local inference. This speed is critical for applications that need to respond instantly without cloud latency.

Cognition integrated NeMo Switchyard into Devin Desktop and achieved near-frontier performance on coding benchmarks while reducing mean cost by 28% relative to routing all requests to a single frontier model. Classmethod reported a 27% cost reduction while maintaining quality using Switchyard internally. These aren't marginal improvements; they're the kind of efficiency gains that make the difference between a profitable AI application and one that's too expensive to scale.

Where Can You Run These Models Locally?

Nemotron 3.5 Lightning is available across multiple platforms and deployment options. You can run it on NVIDIA RTX PCs, NVIDIA DGX Spark systems, NVIDIA DGX Station, and NVIDIA Jetson edge devices. It's also available through Hugging Face, ModelScope, OpenRouter, and as an NVIDIA NIM microservice through a broad ecosystem of cloud partners and inference platforms. NeMo Switchyard is open source and available on GitHub, with integrations coming to partner platforms soon.

The broader ecosystem is expanding rapidly. NVIDIA's DGX and RTX platforms now support additional open models including Cosmos 3 Edge, MiniMax-H3, Poolside's Laguna S 2.1, and DeepSeek-V4-Flash. LTX-2.5, a new video generation model, delivers 2 times performance improvement and 40% memory savings on NVIDIA RTX GPUs and DGX systems, making high-quality video generation practical on local hardware.

The shift toward local, routed inference represents a fundamental change in how enterprises deploy AI. Instead of treating AI as a cloud service, organizations are building hybrid systems that keep sensitive work local, route tasks intelligently to minimize cost, and maintain full control over their data and models. For teams managing large-scale agentic AI systems, this approach can reduce costs by 27% to 74% depending on workload and routing strategy, while actually improving response times and privacy.