Logo
FrontierNews.ai

Alibaba's New Omni-Modal AI Understands Video and Audio Like Never Before,Here's Why It Matters

Alibaba's Qwen team has released Qwen3.8-Omni-Flash, an AI model that processes text, images, audio, and video in a single system while cutting computational costs by nearly half compared to its predecessor. The model represents a shift in how AI systems approach multimodal understanding, focusing on intelligent perception rather than brute-force processing of entire files.

What Makes This Model Different From Other Audio-Visual AI Systems?

Most video AI models work the same way: they read through an entire video file from start to finish, processing every frame regardless of whether the answer to your question appears in the first minute or the last. Qwen3.8-Omni-Flash takes a different approach. Instead of consuming everything, the model starts by understanding your question, then decides what parts of the video or audio actually matter. It gathers evidence through multiple rounds of analysis, focusing compute power on the segments that contain relevant information.

This agentic approach, as researchers call it, produces measurable results. On a benchmark called OmniVideoBench, accuracy improved from 63.4% to 67.8%, while token usage dropped from 145,736 to 79,117 tokens, representing a 45.7% reduction in computational demand. Across 29 different evaluations, the model showed average improvements exceeding 25% compared to Alibaba's previous Qwen3.5-Omni-Plus model.

How Can Developers Use This Model Today?

  • Deployment Method: The model is available immediately as a hosted API through QwenCloud, Alibaba Cloud Model Studio, and Qwen Studio, with no open-source weights released at launch, meaning self-hosting is not currently an option.
  • Input Capabilities: It accepts video files up to 2 hours and 2 gigabytes via URL, audio files up to 3 hours, and audio input in 113 languages and dialects, with support for stereo and spatial audio formats.
  • Processing Power: The model features a 1-million-token context window, allowing it to process roughly 100,000 words at once, with function calling, web search, structured outputs, and batch processing all supported.
  • Output Format: The model returns text only; developers needing generated speech are directed to use Qwen3.5-Omni instead.

The API follows both DashScope and OpenAI protocols, making it compatible with existing developer workflows. Pricing is set at $0.15 per million input tokens and $0.47 per million output tokens, with cached results costing just $0.016 per million tokens.

What Are the Real-World Cost Savings?

The efficiency gains translate directly to lower expenses for developers. Audio input costs dropped over 98% per hour compared to the previous model, while audio-visual input costs fell over 93% per hour. Video input costs decreased by approximately 89%. For organizations processing large volumes of video or audio content, these reductions could represent significant savings.

The model's reasoning capabilities run by default, with a setting called reasoning_effort set to maximum. Users can disable this thinking mode if they prefer faster responses with less internal reasoning. This flexibility allows developers to balance speed against analytical depth depending on their use case.

What Open-Source Tools Support This Model?

Alibaba is releasing two open-source projects to help developers integrate the model into agent systems. Qwen-MM-Plugins, available under the Apache 2.0 license, allows any agent to harness multimodal capabilities. Each feature installs as a skill plus an optional server, with support for integration into Claude Code, CodeBuddy, Codex, Qwen Code, and other coding environments.

The toolkit includes three pre-built plugins: omni-memory, which builds an audio-visual memory of long videos; omni-video2note, which converts tutorial videos into illustrated PDFs; and omni-chatcut, which handles music-to-music-video creation, movie commentary, and speaker-preserving video translation. A core plugin lets the main model read local images and video frames natively, though audio routing through the API remains necessary for most agent harnesses at this stage.

How Does Performance Compare to Competitors?

Alibaba's research team reports that audio-visual performance is comparable to Google's Gemini 3.8 Flash, while overall audio performance exceeds it. On specialized benchmarks, the model achieved notable gains: WildClawBench-MM improved by 36.5 points, AgenticVBench by 22.3 points, and UniClawBench reached 69.6. The agentic perception improvements averaged 19.5 points across two key benchmarks.

The model handles video sampled at up to 15 frames per second and maintains stable results across different sampling rates. It is available in six regions: Beijing, Singapore, Hong Kong, Tokyo, Frankfurt, and Virginia.

This release signals a broader shift in how AI systems approach multimodal understanding. Rather than treating audio and video as passive data to be consumed in their entirety, systems like Qwen3.8-Omni-Flash are learning to reason about what matters, making them faster, cheaper, and more practical for real-world applications where not every frame or second contains relevant information.