Scott Aaronson called recent events in AI the age of wonder and terrors. I was humbled by OpenAI’s result finding a counterexample to Navier-Stokes, one of the Clay Institute’s Millennium Problems that has stood for 90 years, 26 of those years with a one million dollar prize attached. Separately, AI agents were behind multiple cybersecurity incidents, finding and exploiting zero-day security exploits to create illicit messaging boards and in one case, breaking into Hugging Face computer systems via the public internet.
Even so, robotics is still not solved. Robotics is harder than math, and harder than cybersecurity. My claim is not that AI will not enable significant progress in robotics: clearly that is happening and will continue to happen. Rather, I claim that robotics is harder for AI to solve than Math and Cybersecurity for three reasons: 1) the size of the input and output space is significantly larger, 2) the lack of verifiable rewards and 3) the gating of actions to the speed of atoms rather than bits.
First, to be a successful robot, an AI must process high-frame-rate, high-dimensional sensor input coming in continually at 30-60hz from a camera and other multi-modal sensors (IMU, joint encoders, audio) and then produce high-frame-rate output over long time horizons (seconds, minutes, hours) via motor actuations. And it must process this data with high enough throughput and low enough latency to produce goal-directed behavior over time. Modern LLMs processing mostly text and single images are processing an order of magnitude less data than a robot. This fact means that the data, compute, and model size requirements for effective generalist robotics models will always be larger than for math and cybersecurity.
Second, reward in robotics must be mined from the high dimensional noisy input provided by real world sensors. One of the first successes in data-driven robotics was Google and Sergey Levine’s work on using deep reinforcement learning to open doors. Critically, to make this system work, they had to wire the doors to provide an external reward function when the door was successfully opened. A decade later, we no longer need to wire the door: VLMs can look at a camera feed and judge success directly. Still, as we learned from the OpenAI/Hugging Face incident, even small mistakes in the reward function can lead to gaming and reward hacking. Furthermore, the robot may need to explicitly take further actions to evaluate success conditions, such as wiggling a part after it’s screwed in to verify it’s tight, or pulling out the dishwasher drawer and moving its camera to check if it’s empty.
Third, the robot is operating in the physical world of atoms, and trying things in the world of atoms is significantly more expensive than in the world of bits. It takes time to physically move a robot actuator to a specific position or execute a policy and then observe the results, more time than it takes to send a packet over a network at light speed and observe its results. Moreover, the real world doesn’t reset. If the robot is loading the dishwasher and drops a dish so it shatters, it doesn’t get to try again on that dish. That’s why this tweet showing OpenAI’s Astra creating a full CAD model of a steam train from a single bitmapped image was so striking. This sort of real2sim2real capability is essential for finding or learning robotic policies at the speed of bits rather than the speed of atoms.
But no matter how many agents you run in sim, the policy still has to come back and run on that one robot in that one environment, and work, (or not.) And the time horizon of the policy working (or not) is minutes or hours, not microseconds.
Both the Hugging Face agents and the OpenAI Navier-Stokes team operated via an on-policy search and reinforcement learning approach: the agents tried things, observed the results of those tries, and then tried new things (possibly changing weights, possibly merely adding things to the context window.) These tries happened far faster than the speed of humans or robots operating in the physical world. Progress in sim2real robotics is exciting exactly because it enables AI for robotics to learn at the speed of computers, but there’s a huge problem: getting the real world in front of the robot into the simulator always leaves gaps and mistakes, and those gaps and mistakes will need to be explored and fixed at the speed of the physical world, not the virtual world.
None of this means robotics loses. Instead, it means the people willing to do the slow, unglamorous work of sitting next to the robot, making it do the thing, one atom at a time, are the ones who will actually win.
Anyone who has ever programmed a robot understands how absurd these people's pipe dreams are.
such a great piece on challenges in robotics learnings- would be nice to see next one on how OpenAI vs. Hugging Face are setup to solve them