A new benchmark framework called SemComp-Bench has exposed a fundamental gap in how AI video generators are evaluated: most systems can produce visually impressive videos but fail to reliably complete instructed tasks while preserving details from reference images. The research reveals that this blind spot affects both commercial tools and open-source models, potentially reshaping how enterprises choose video generation systems.

What Problem Are Researchers Actually Trying to Solve?

Imagine asking an AI video generator to fold a specific banknote into origami shaped like a turtle. You'd expect the result to show that exact banknote being transformed, not just any random banknote appearing in the video. That distinction matters enormously for commercial applications, yet it's rarely measured by existing benchmarks.

The problem isn't about visual quality or smoothness. Modern video generation models produce visually impressive results with strong temporal coherence. What they struggle with is something more fundamental: achieving an instructed outcome while maintaining semantic grounding, which means keeping the task-relevant details from the original image intact.

Existing benchmarks for video generation focus on visual fidelity, temporal coherence, and how well the AI follows text instructions. They measure whether the video looks smooth, whether objects move realistically, and whether the final result matches the description. But they rarely assess whether the AI actually completes the task while preserving the semantic relationship between the reference image and the outcome.

How Does the New Benchmark Actually Work?

Researchers introduced SemComp-Bench, an evaluation protocol that uses a vision-language model, or VLM (an AI system trained to understand both images and text), to answer structured binary questions about whether videos achieve their intended goals. The framework measures performance across two complementary dimensions:

• Outcome Achievement Score: Measures whether the video realizes the intended outcome and maintains semantic grounding, with safeguards for task-relevant entity consistency and global visual continuity.
• Generation Reliability Score: Assesses physical violations, blur, rendering artifacts, local instability, and corrupted text or interface elements that might undermine the final result.
• Evidence-Grounded Evaluation: Uses a vision-language model to answer structured binary questions about the video, with each answer supported by visual evidence for focused and interpretable judgments.

To build this benchmark, researchers constructed SemComp-Data, an evaluation dataset covering six different domains with hundreds of image-text-video triplets. Each instance includes a reference image, detailed and brief instructions describing the intended outcome at different levels of specificity, and an outcome-centric video clip showing successful task completion.

The curation process involved four stages: filtering raw videos to identify those with visually observable outcomes, mining the exact frames showing the reference state and completed outcome, extracting outcome-focused video clips, and structuring detailed instructions that describe the intended result.

Why Should Creators and Enterprises Care About This?

The implications are significant for anyone relying on AI video tools for commercial production. If an AI video generator can't reliably preserve the specific object or context you provide while completing a task, the results may look polished but fail to deliver what you actually requested. A marketing team might ask an AI to transform their specific product packaging into a holiday-themed design, only to receive a video showing a generic package instead.

The research shows that achieving intended outcomes while maintaining task-relevant semantic grounding in reference images remains challenging across representative video generation models. This includes both closed-source systems such as Pika, Sora, Veo, MovieGen, Runway, and Kling, as well as open-source alternatives like Wan2.2, CogVideoX, HunyuanVideo, and Pyramid Flow.

How to Evaluate AI Video Tools for Task Completion

When selecting or testing an AI video generation system, consider these practical evaluation criteria:

• Reference Preservation: Test whether the tool maintains the specific object, person, or context you provide in the reference image, rather than substituting it with a generic alternative.
• Outcome Verification: Check whether the generated video actually completes the instructed task, not just produces a visually similar result that ignores your source material.
• Semantic Consistency: Verify that task-relevant details remain consistent throughout the video, ensuring the transformation or action applies to your specific reference material.
• Quality Beyond Aesthetics: Look beyond smooth motion and visual polish to assess whether the tool delivers on the actual task you requested.

The SemComp-Bench framework provides a standardized way to identify and measure these gaps before they impact production workflows. By establishing clear criteria for outcome achievement and semantic grounding, researchers have created a tool that could help enterprises and creators make more informed decisions about which video generation systems actually deliver on their promises.

As AI video generation becomes increasingly central to commercial production, the gap between visual quality and task completion becomes more consequential. This research suggests that the next generation of AI video tools will need to focus not just on making videos that look impressive, but on making videos that reliably complete the specific tasks users request while respecting the source material they provide.