If you think an artificial intelligence model running on thousands of cutting-edge computer chips is smart, allow me to introduce you to the concept of a 1-year-old.
OK, so babies might not be able to write computer programs, solve advanced math problems, or debate philosophical ideas. But unlike today’s AI models, which consume an ocean’s worth of training data and as much energy as a small country, babies learn to make sense of the world with amazing efficiency. They identify new objects after seeing them once or twice, and they learn through fleeting observation and physical interaction.
When it comes to improving AI, babies—and the architecture of their brains—might hold crucial insights. Building a more baby-like version of AI could make frontier models less costly and less energy intensive, and it might also be valuable if AI-powered robots are to learn about their environments in a more natural way.
To explore this bold new frontier, researchers at Meta, Stanford University, the University of Tokyo, and France’s École Normale Supérieure developed a new test that highlights the learning skills of babies and pushes AI researchers to design algorithms that match them.
The EgoBabyVLM Challenge judges how well vision language models, or VLMs, which learn from both text and imagery, can make sense of the world as a baby sees it. It requires a model to describe the world after ingesting about a thousand hours of video collected from cameras strapped to the heads of infants and toddlers. (Yes, really.)
It turns out that the cutting-edge models fail miserably when fed this realistic and messy footage, which suggests there may be something different about the design of the baby brain that enables it to learn so rapidly from so little information.
Instead of curated datasets, babies learn from a kaleidoscopic view of things: parents talking about objects that are no longer visible, indicating things using their gaze or a gesture, or discussing events from the past or in the future rather than whatever’s happening right then. Babies learn not just from language but also from a rich multimodal and tactile experience, says Michael Frank, a cognitive scientist at Stanford University who specializes in language learning and was involved with EgoBabyVLM’s development.
The test shows that when it comes to AI, “it’s clear that there’s more [than just language] that’s needed,” Frank says.
Language Learning
EgoBabyVLM is just the latest example of how scientists are using AI to explore human intelligence. A challenge called BabyLM, introduced in 2023, tasked AI models with learning the syntax of language using about the same amount of data a 10-year-old takes in—tens of millions of words, compared to trillions for AI models. Remarkably, it turns out that transformer-based AI models—which process language by paying attention to the relationship between words across different sentences—can do this quite well, a finding that challenges Noam Chomsky’s ideas concerning how syntax may be hardwired into the human brain.
Ryan Cotterell, a linguist at ETH Zurich who first developed BabyLM, says the situation is different when it comes to understanding the physical world. “There isn't going to be a large corpus of human interactions—there's no internet of human interactions,” he says.
Joshua Tenenbaum, a cognitive scientist at the Massachusetts Institute of Technology, notes that BabyLM showed models do not acquire “common sense” about the physical world, social dynamics, or theory of mind.
“Transformers are very good at finding patterns in data,” says Tenenbaum. “But it does seem that just pure pattern learning systems are not able to take the kind of data that a baby or a child receives and learn all the things that they do.”
An enduring question is whether evolution found a way to optimize certain learning skills in humans and other animals, or if simple learning algorithms can do everything we do. “There is a lot of debate in cognitive science and neuroscience about how much is built into the brain evolutionarily,” Tenenbaum says. “The brain is incredibly complex, and there's a lot of built-in structure and architecture.”
In 2024, researchers showed that a basic VLM can learn simple things, like what a ball is, purely by consuming data recorded from the head of a single infant. But this is a ways away from reasoning about the world in sophisticated ways. “The mystery is how children get to the full capabilities that they have even at the age of 2,” says Brendan Lake, a cognitive scientist at Princeton University who was involved with the project.
The authors of the EgoBabyVLM paper suggest that borrowing different ideas from cognitive science and neuroscience could enable progress toward more humanlike learning algorithms. This includes designing models that can pay attention over longer periods and can interpret social cues.
Stanford’s Frank has already shown that novel approaches can get us closer to baby-like AI. Earlier this year, he and colleagues tested a new kind of model that’s adept at learning causality and visual and temporal relationships—or how objects affect one another over time—using the same baby-head video data. They found the new model was able to learn about the dynamics of different objects, a foundation for physical reasoning, much more effectively.
It’s a tantalizing possibility: Perhaps models that are biased to learn more rapidly about things like physics and social relationships could be more efficient learners overall.
“EgoBabyVLM is a wonderful challenge,” says Lake. “I'm excited to see what kinds of new architectures, approaches, and ingredients researchers come up with.”
This is an edition of Will Knight’s AI Lab newsletter. Read previous newsletters here.
Instagram Engineering
Shopify Engineering