A new research framework called CaP-X is demonstrating that vision-language models can enable robots to generate and execute their own control code, achieving performance comparable to human experts on complex manipulation tasks. Rather than relying on pre-programmed instructions or learning from massive datasets, robots using this "Code-as-Policy" approach can synthesize executable programs by combining perception and control primitives, fundamentally changing how embodied AI systems are designed.

How Are Vision-Language Models Changing Robot Control?

Traditionally, robots have been controlled through explicit programs written by human engineers, combining perception, geometry, planning, and feedback. This classical approach offers strong interpretability but requires extensive manual effort and produces task-specific solutions that struggle to generalize. The emergence of Vision-Language-Action (VLA) models, which learn from large-scale visuomotor datasets, promised to overcome these limitations but introduced new problems: they lack interpretability, struggle with environmental changes, and require retraining for new robot embodiments.

CaP-X bridges these two paradigms by using modern coding agents to replace the human engineer. The framework includes four core components designed to systematically evaluate how well language and vision-language models can control robots through code generation. Researchers tested 12 state-of-the-art models across varying levels of abstraction, interaction, and perceptual grounding to understand what actually drives robot performance.

What Did the Research Reveal About Model Capabilities?

The findings expose a critical dependency: task success rates improve significantly when models have access to human-crafted abstractions, but performance degrades substantially when these design scaffolds are removed. This reveals that much of the apparent capability in prior Code-as-Policy systems came from the structure provided by designers, not purely from the models themselves.

However, the research also discovered that scaling test-time computation can substantially improve robustness even when agents operate over low-level primitives. By combining multiple techniques, researchers achieved near human-level reliability on several manipulation tasks in both simulation and real robot embodiments.

Ways to Improve Robot Coding Performance

• Multi-turn Interaction: Allowing robots to iteratively debug and refine their generated code across multiple attempts, rather than relying on single-shot program generation, significantly improves success rates on complex tasks.
• Visual Differencing: Providing robots with visual feedback that highlights differences between intended and actual outcomes helps ground task-relevant visual features into code generation more effectively.
• Automatic Skill Synthesis: Building task-agnostic skill libraries that robots can automatically compose reduces the need for human-designed primitives while maintaining performance on diverse manipulation tasks.
• Ensembled Reasoning: Running multiple code-generation models in parallel and combining their outputs creates more robust solutions than any single model could produce alone.

The research team developed CaP-Agent0, a training-free framework that combines these techniques. Remarkably, this approach achieved performance comparable to human expert baselines without requiring any machine learning training, suggesting that the right combination of test-time computation and multimodal grounding can compensate for lower-level interfaces.

CaP-X also supports reinforcement learning through CaP-RL, which uses verifiable rewards to improve success rates. Importantly, programs synthesized through this approach transfer directly to real robots, addressing a long-standing challenge in robotics where simulation-trained systems often fail in physical environments.

Why This Matters for Enterprise AI and Beyond

The framework provides an open-access platform for advancing embodied coding agents, integrating 187 tasks from standard robot manipulation simulators under a shared primitive design compatible with both simulation and physical systems. This systematic approach to benchmarking and improving vision-language models in robotics comes at a time when the broader AI research community is intensifying focus on multimodal systems.

Vector Institute researchers are contributing significantly to this momentum, with 73 accepted papers at the International Conference on Machine Learning (ICML) 2026, taking place July 6 through 11 in Seoul, South Korea. Vector's research portfolio spans reinforcement learning, generative AI, multimodal and vision-language systems, autonomous agents, and planning, demonstrating the breadth of work advancing embodied AI and responsible deployment of agentic systems.

The implications extend beyond robotics. As vision-language models become more capable at grounding abstract reasoning in visual perception, the ability to generate executable code that bridges perception and action opens new possibilities for autonomous systems across manufacturing, logistics, and scientific research. The CaP-X framework's emphasis on systematic evaluation and the removal of designer scaffolding provides a template for honestly assessing what multimodal models can actually do versus what they appear to do when given significant structural support.