Allow me to introduce you to the robot cat butler I’m developing with help from AI.
Using OpenAI’s GPT-6 Sol, I taught a virtual version of Microduck—a cute robot from Pollen Robotics and Hugging Face—to walk around a room and identify any bengal cats in its vicinity. (The clip above shows the robot waddling toward a target feline; its eye view is shown top right.)
I’m still waiting patiently for a physical Microduck—please hurry up and ship it, guys—but simulated behavior is relatively simple, so it should transfer over to the real thing. I plan to have the robot automatically dispense a treat for my pet bengal, Leona, perhaps by bowing its head to release one from a 3D-printed, head-mounted storage chamber.
I was able to code and train the virtual robot in just a few hours, which is, quite frankly, incredible. Previously, this kind of thing might have taken me weeks to figure out. It was possible because OpenAI’s latest models are remarkably good at understanding and manipulating 3D representations of the physical world and then translating that into useful code, agentic actions, and robot training.
Real roboticists are using the latest models to do some amazing stuff. When OpenAI released Astra this past September, University of California, Berkeley professor Ken Goldberg and his students saw demos that showcased its ability to render 3D images. They wondered if it could help with the simulations required to train different kinds of robots. These simulations need to be incredibly detailed and extremely accurate to ensure that sophisticated behavior—a humanoid that can dance or a robot hand that can grasp—will transfer faithfully to the real world.
One member of Goldberg’s lab gave Astra access to an Nvidia tool called Newton, showed it a video clip of a real robot in action, and asked if it could create a simulation of the scene.
“What blew us all away was that within an hour it had produced a full 3D simulation with all the parameters set—the position of the camera, the lighting, the background, all the materials,” Goldberg tells me.
Astra is also able to automate much of the drudgery involved with training a robot in simulation, Goldberg says, by running experiments and examining the results.
This is precisely what I found while training the virtual Microduck. Along with GPT-6 Sol, I used an agentic harness from Nous Research called Hermes and a Nvidia DGX Spark, a personal computer that contains a powerful AI chip.
I’ve previously written about using agents and coding models to configure, program, and debug physical robots. Normally, this kind of work requires significant expertise; modern coding models make it a cinch. But OpenAI’s newest model went far beyond just writing code.
It used a piece of Nvidia software called Isaac Sim to feed instructions to a model that controlled the virtual robot, watch videos of it in action to assess how well things were working, and suggest tweaks to its training regime.
This process allowed me to train the Microduck to walk around without toppling over. I initially tried getting GPT-6 Sol to teach the robot to moonwalk—I fed it a GIF of Michael Jackson sliding across the stage, which it tried to copy—but after hours of effort, it only resulted in a very jaunty backward jog. (The robots don’t have us totally beat quite yet.)
Jim Fan, a researcher at Nvidia who has collaborated with Goldberg, says the way AI can now control robots mirrors how software agents function. LLMS can use models trained to control physical actions—visual language action (VLA) models—just as they access different software on a computer.
This week, Fan, Goldberg, and others open sourced an agentic framework for robots called Aspire, which allows a robot to discover and use new skills. One example: GPT-6 Astra figured out how to use a pair of robot arms to solve a Rubik’s cube.
“It picks it up, tries to twist it, but has the wrong angle,” Fan says. “Then it reasons for a very, very long time, but it eventually is able to solve it because it learns the skills over time.”
At the Conference on Robot Learning (CoRL), held in Austin, Texas, this week, Fan gave a keynote speech in which he talked about the incredible realization that AI models can automate a lot of the work needed to have a robot do sophisticated things. Fan showed off a demo in which Astra was put in control of a robot arm and told to paint a picture of the Golden Gate Bridge. Using some paint and brushes nearby, it duly completed the task.
Of course, getting robots to work reliably and efficiently is still extremely difficult, requiring significant engineering expertise. The vision system on my robot duck does not work well enough to be deployed in a real product.
Still, the ease and speed of training robots with AI could soon have an enormous impact on manufacturing and e-commerce, and Goldberg says a number of industrial robotics and automation firms have shown an interest in the approach.
Much like vibe coding a few years ago, we may well see breakthroughs in “vibe robotics” as hobbyists and engineers realize how easy it is to program and train robots. And, assuming I can perfect my robot helper, Leona may never have to go without treats again.
This is an edition of Will Knight’s AI Lab newsletter. Read previous newsletters here.
Instagram Engineering
Shopify Engineering