NVIDIA has expanded its Agent Toolkit to include Omniverse libraries, giving artificial intelligence systems the ability to work directly with 3D environments, sensor simulations, and physics engines. This integration marks a significant step toward automating the preparation of virtual worlds where robots and autonomous systems are trained before real-world deployment.

Why Does 3D Simulation Matter for AI Agents?

Robots, factories, and autonomous systems are typically designed and tested in simulation environments before they operate in the real world. However, preparing 3D content for these simulations has traditionally required significant manual work. Engineers must define materials, scale, physical properties, sensors, and labels on digital assets. With Omniverse libraries now integrated into the Agent Toolkit, AI agents can automate much of this preparation work.

The toolkit now includes three core capabilities that extend AI agents into physical simulation workflows:

• RTX Sensor Simulation (ovrtx): Allows AI agents to generate camera, lidar, radar, and other sensor outputs from 3D scenes, helping developers and agents evaluate how physical AI systems perceive virtual environments.
• GPU-Accelerated Physics (ovphysx): Applies realistic behavior to 3D objects using properties such as collisions, mass, friction, and motion, enabling teams to test how objects and systems interact in simulation.
• CAD-to-SimReady Conversion: Transforms computer-aided design data into simulation-ready assets built on OpenUSD, adding the properties needed for physical AI simulation and virtual testing.

These tools are now available on GitHub, and NVIDIA has published a blueprint for integrating Omniverse libraries directly into Blender, a widely used 3D creation software.

Which Companies Are Already Using This Technology?

Major software makers and startups have begun adopting these Omniverse libraries as part of the expanded Agent Toolkit ecosystem. SideFX, known for its Houdini procedural 3D software, is using the ovrtx and ovphysx libraries to explore how agents can help technical artists generate content, test physics, and prepare assets for simulation. PTC, which makes the Onshape cloud-native CAD platform, is using OpenUSD and RTX sensor simulation to connect design workflows with physical simulation capabilities.

Smaller companies are also experimenting with these tools. Palatial is using Omniverse CAD-to-SimReady skills to automate the creation and validation of simulation-ready assets from CAD inputs. Lightwheel is leveraging Omniverse Content Agents to generate physically accurate assets from text prompts through its SimReadyGen technology. ForgeCAD and Moonlake AI are exploring agent-driven 3D content workflows that use Omniverse capabilities to generate, augment, and prepare assets for physical AI simulation.

How to Deploy Agent Toolkit Locally Without Cloud Dependency

One of the key advantages of the expanded toolkit is the ability to run AI agent workflows on local hardware without requiring internet access. NVIDIA has outlined a complete local AI agent stack for its DGX Station systems, which are high-performance computing machines designed for AI workloads.

• Local Deployment: The Agent Toolkit on DGX Station combines NemoClaw, the Nemotron 3 Ultra open-source language model, Omniverse libraries, and a secure runtime on a single local machine without cloud connectivity requirements.
• Scalability: Users can link multiple DGX Station systems together as their workloads grow, allowing organizations to scale their agent infrastructure while maintaining data control and privacy.
• Ecosystem Support: LangChain, Nous Research, and OpenClaw are among the organizations tuning or using this local agent stack for their own workflows, indicating broad adoption across the AI agent development community.

DGX Station systems are available to order from ASUS, Dell Technologies, Exxact, GIGABYTE, HP, MSI, and Supermicro. For developers working with edge devices, NVIDIA also released Cosmos 3 Edge, a 4-billion-parameter model designed for deployment on Jetson, RTX Pro, and GeForce RTX hardware, enabling on-device vision analytics and robot action in real time.

What Does This Mean for the Future of Physical AI?

The integration of Omniverse libraries into the Agent Toolkit represents a broader strategy by NVIDIA to bring AI models and agents closer to production tools, edge devices, and local infrastructure. At SIGGRAPH, a major graphics and simulation conference, NVIDIA demonstrated a "SimReady" Blender workflow that shows how software makers can add agent-ready simulation capabilities, including RTX sensor simulation, physics, and validation, into existing 3D applications while keeping creative control in the hands of human designers.

This approach addresses a critical gap in the AI agent ecosystem. Rather than requiring agents to operate in isolation, these tools allow agents to work within the existing software that designers, engineers, and technical artists already use daily. The workflow can run locally on RTX-powered systems such as ASUS, Dell, HP, Lenovo, and Microsoft Surface devices, which are scheduled to be available this fall, or on more powerful NVIDIA DGX Station systems available now.

The breadth of adoption suggests that automating 3D asset preparation and simulation workflows is becoming a priority across multiple industries. Whether in robotics, autonomous systems, or factory automation, the ability for AI agents to independently prepare and validate 3D environments could significantly reduce the time and expertise required to move from virtual testing to real-world deployment.