Vision language models, or VLMs, which learn from both text and images, perform surprisingly poorly when trained on video footage from cameras mounted on babies' heads, suggesting that human infants possess learning mechanisms that today's most advanced AI systems have yet to replicate. Researchers from Meta, Stanford University, the University of Tokyo, and France's École Normale Supérieure developed the EgoBabyVLM Challenge to test how well these cutting-edge models could understand the world as a baby sees it, using roughly a thousand hours of egocentric video data collected from infants and toddlers.

What Makes Baby Learning So Different From AI?

Babies learn with remarkable efficiency compared to modern AI systems. A typical infant can identify new objects after seeing them just once or twice and learns through fleeting observation and physical interaction, all while consuming far less data and energy than the largest AI models. In contrast, today's frontier AI models require an ocean's worth of training data and consume as much electricity as a small country to achieve their capabilities.

The key difference lies in how babies absorb information from their environment. Rather than relying on curated, clean datasets, infants learn from a kaleidoscopic view of the world: parents talking about objects that are no longer visible, people indicating things through gaze or gesture, and discussions about events from the past or future rather than just what's happening in the present moment. Babies also learn from a rich multimodal and tactile experience that goes far beyond what current VLMs can process.

"It's clear that there's more that's needed," said Michael Frank, a cognitive scientist at Stanford University who specializes in language learning and was involved with EgoBabyVLM's development.

Michael Frank, Cognitive Scientist at Stanford University

How Are Researchers Testing AI Against Baby Intelligence?

The EgoBabyVLM Challenge represents a bold new frontier in AI research, pushing vision language models to describe the world after ingesting video collected from cameras strapped to the heads of infants and toddlers. When exposed to this realistic and messy footage, cutting-edge models fail dramatically, suggesting that something fundamental about the baby brain's architecture enables rapid learning from minimal information.

This challenge builds on earlier work in the field. A previous challenge called BabyLM, introduced in 2023, tasked AI models with learning language syntax using roughly the same amount of data a 10-year-old absorbs, which is tens of millions of words compared to the trillions used to train modern AI systems. Remarkably, transformer-based AI models, which process language by analyzing relationships between words across sentences, performed quite well at this task, challenging long-held assumptions about how human language learning works.

However, the situation differs dramatically when it comes to understanding the physical world. Ryan Cotterell, a linguist at ETH Zurich who developed BabyLM, noted that "there isn't going to be a large corpus of human interactions; there's no internet of human interactions," making it harder for AI to learn physical reasoning the way humans do.

What Gaps Remain in AI's Understanding?

Research shows that current AI models struggle with several critical aspects of human learning that babies master naturally. Joshua Tenenbaum, a cognitive scientist at the Massachusetts Institute of Technology, explained that BabyLM demonstrated models do not acquire "common sense" about the physical world, social dynamics, or theory of mind, the ability to understand that others have beliefs and desires different from one's own.

• Physical Reasoning: Transformers excel at finding patterns in data but cannot learn the kind of physical intuition that allows babies to understand how objects interact and affect one another over time.
• Social Understanding: Current VLMs lack the ability to interpret social cues and understand human interactions in the nuanced way that infants develop naturally.
• Causal Learning: AI models struggle to understand causality and temporal relationships, which babies grasp through observation and interaction with their environment.

"Transformers are very good at finding patterns in data, but it does seem that just pure pattern learning systems are not able to take the kind of data that a baby or a child receives and learn all the things that they do," said Joshua Tenenbaum.

Joshua Tenenbaum, Cognitive Scientist at Massachusetts Institute of Technology

How Are Researchers Designing Baby-Like AI?

Scientists are exploring whether evolution optimized certain learning skills in humans and animals, or whether simple learning algorithms could theoretically do everything humans do. This fundamental question drives new research into more efficient AI architectures inspired by cognitive science and neuroscience.

In 2024, researchers demonstrated that a basic VLM could learn simple concepts, like what a ball is, purely by consuming data recorded from a single infant's head camera. While this represents progress, it remains far from the sophisticated reasoning that even two-year-old children demonstrate.

Stanford's Michael Frank has already shown that novel approaches can move us closer to baby-like AI. Earlier this year, he and colleagues tested a new kind of model adept at learning causality and visual and temporal relationships, or how objects affect one another over time, using the same baby-head video data. The new model learned about object dynamics and physical reasoning much more effectively than standard approaches.

"The mystery is how children get to the full capabilities that they have even at the age of 2," said Brendan Lake, a cognitive scientist at Princeton University who was involved with the project.

Brendan Lake, Cognitive Scientist at Princeton University

What Could Baby-Inspired AI Achieve?

Building more baby-like versions of AI could unlock significant practical benefits. Models that learn more efficiently would be less costly to train and operate, consuming far less energy than current frontier systems. Additionally, if AI-powered robots are to learn about their environments in a natural and adaptive way, they may need to incorporate learning mechanisms inspired by human infant development.

The authors of the EgoBabyVLM paper suggest that borrowing ideas from cognitive science and neuroscience could enable progress toward more humanlike learning algorithms. This includes designing models that can pay attention over longer periods and can interpret social cues more effectively. Perhaps models biased to learn more rapidly about physics and social relationships could become more efficient learners overall.

"EgoBabyVLM is a wonderful challenge. I'm excited to see what kinds of new architectures, approaches, and ingredients researchers come up with," said Brendan Lake.

Brendan Lake, Cognitive Scientist at Princeton University

The EgoBabyVLM Challenge represents a pivotal moment in AI research, shifting focus from raw computational power and massive datasets toward understanding the fundamental learning principles that make human infants such efficient learners. As researchers continue to explore these questions, the insights gained could reshape how we design the next generation of AI systems, making them not just smarter, but fundamentally more efficient and capable of learning from the world around them in ways that more closely mirror human cognition.