Vision language models (VLMs) have crossed a critical threshold: they can now process audio and video as continuous streams rather than waiting for complete prompts. This shift, documented across releases from OpenAI, Meta, and open-source teams in mid-July 2026, represents a fundamental change in how AI systems interact with the world. Instead of analyzing a finished photo or transcript, these models now handle interruptions, silence, corrections, and real-time feedback.

What Changed in Multimodal AI This Month?

The headline story is real-time capability. On July 8, OpenAI began rolling out GPT-Live-1 and its smaller variant through ChatGPT Voice, both supporting full-duplex audio, meaning the model can listen and speak simultaneously without waiting for you to finish talking. The system retains search, image input, file handling, and visual cards, but live video and screen sharing were not available at launch.

Parallel to OpenAI's move, OpenMOSS released three open-weight models called MOSS-VL Realtime under the Apache 2.0 license. These 11-billion-parameter models process timestamped continuous frames and handle the messy reality of conversation: interruptions, silence, and mid-sentence corrections. The weights and inference code are publicly available, though each model instance supports only one session at a time, and the default frame cap is 256.

Meta also made moves in this space, launching Muse Image and previewing Muse Video and Spark 1.1 through its Model API. Spark accepts visual and audio context, including video and PDFs, expanding the types of input these systems can handle. Muse Video remains unreleased, with the team noting audio synchronization and fast motion as current limitations.

How Are Developers and Companies Using These Real-Time Models?

• Production Deployments: NVIDIA published a DeepStream 9.1 tutorial for multi-camera 3D tracking across four or twelve cameras, with source code and sample applications publicly available. This enables real-world applications like warehouse monitoring or sports analytics that require synchronized video from multiple angles.
• Robot Training and Evaluation: GigaWorld-1 uses generated video to evaluate how robot policies perform in learned video worlds, with 1.3 billion and 5 billion parameter Stage-1 weights released under Apache 2.0. LeRobot 0.6 added world-model policies and expanded vision-language-action (VLA) APIs, covering more of the training and evaluation loop for robotics applications.
• Outdoor Surveillance at Scale: Syntiant and PRADCO report more than one million deployed outdoor cameras in service, with on-device models classifying animals and sending selected image or video events. This represents cumulative deployment, not a two-week rollout, but signals real-world adoption of multimodal perception.

Why Does Real-Time Matter for Vision Language Models?

The shift from batch processing to streaming changes what these systems can do. A VLM that waits for a complete image or transcript is fundamentally different from one that processes frames as they arrive and responds mid-conversation. Real-time systems enable natural dialogue, faster decision-making in robotics, and the ability to handle interruptions without restarting the entire inference pipeline. This is especially important for applications like customer service, robot control, and live video analysis where latency and responsiveness matter.

The open-source releases are significant because they democratize this capability. MOSS-VL Realtime's Apache 2.0 license means researchers and smaller companies can experiment with real-time multimodal systems without licensing restrictions. However, there are practical constraints: the models require substantial hardware, and each instance handles only one session, which limits deployment scenarios.

What About Larger and More Specialized Models?

The multimodal landscape is also expanding in parameter count and specialization. Thinking Machines released Inkling with 975 billion total parameters, though only 41 billion are active during inference. It handles text, image, and audio input with a 1 million token context window. However, the model is computationally demanding: minimum footprints are estimated near 2 terabytes and 600 gigabytes depending on precision, making it accessible only to well-resourced teams.

NVIDIA released Audex, a family of models for speech and general audio tasks, with dense 2 billion and sparse 30 billion-total, 3 billion-active variants. These can output text or audio tokens, expanding the modalities that multimodal systems can handle. However, NVIDIA's OneWay license and XCodec2 restrictions limit use to noncommercial work, a constraint that may slow adoption in industry.

Video generation is becoming controllable rather than purely generative. MIRA generates four synchronized Rocket League views with a 5 billion parameter diffusion transformer, achieving 20 frames per second on a single B200 GPU. ProxyPose converts a single selected pixel into a proxy video and recovers a 6-degree-of-freedom camera trajectory, enabling precise camera control from minimal input. These advances suggest that video is shifting from a creative output to a controllable simulation medium.

What Are the Practical Limitations?

Despite the progress, real-time multimodal systems face real constraints. OpenAI's GPT-Live did not launch with live video or screen sharing, suggesting these features require additional engineering work. MOSS-VL Realtime's single-session-per-instance design limits scalability for multi-user applications. GigaWorld-1's Stage-2 weights and reinforcement-learning post-training remain unreleased, meaning the full system is not yet available. Licensing fragmentation is also a problem: some models use Apache 2.0, others use noncommercial licenses, and some have unclear licensing, making it difficult for companies to plan deployments.

The hardware requirements are substantial. Inkling's compressed release needs 600 gigabytes of memory, and Wan-Dancer, a music-conditioned dance video generator, was tested on eight A800 80-gigabyte GPUs without published inference speeds, suggesting it is not yet optimized for efficiency. These constraints mean real-time multimodal AI remains concentrated in the hands of large labs and well-funded companies.

The shift toward real-time, controllable multimodal systems marks a maturation of the field. Rather than treating audio, video, and images as static inputs to be analyzed, the latest models treat them as streams to be processed, interrupted, and controlled. This opens new applications in robotics, live communication, and interactive media, but it also raises questions about scalability, licensing, and who gets access to these powerful systems.