Really cool article - I loved it! This gives words to something I’ve been thinking about for a while (though in my head I used the example of physics and theories of light before Einstein to serve as an analog to modern robotics).
Two things I find interesting after reading the conclusion:
1. Perhaps we’re waiting for a Lavoisier, but perhaps we also simply don’t have enough evidence/data currently to even approach the right theory (I imagine Lavoisier didn’t come up with the building blocks of the periodic table merely from Priestleys oxygen experiment, but rather from a whole bunch of other empirical evidence; perhaps we don’t have that “critical mass” yet?). Curious what you think about this.
2. It is interesting that we’re making seemingly rapid progress on engineering and very little understanding/science progress. Do you think engineering progress is fundamentally limited by science progress? I’d probably be inclined to say yes to this, but as a counterpoint, it feels like we’ve made relatively little progress on the science of deep learning for instance, but we’ve made very impressive empirical progress in building vision and language systems that really work and do useful things.
Re engineering vs science, I think they both feed off each other, and figuring out Stuff That Works is an essential part of progress. But we can hope for a distillation of the Stuff That Works into something more fundamental and explanatory - that is our missing paradigm.
One of my buddies when I was a postdoc empirically found an approach to RL that worked, and wrote a paper about it. Then, he did some math and derived it more formally. This revealed that his empirical update rule was missing a factor in the denominator. Adding the factor the theory said to add improved performance empirically. It doesn't always work out that way, but that was a beautiful example of the duality at work. Maybe I will go dig up the citations and write a post about it. :-)
I think this idea also connects to your first point - I think it was important that Lavoisier and Priestley and others were doing the same experiments, playing with the same things, and telling stories about what they were observing. And then those stories suggested further empirical things to try. Telling stories, to other people (including the stories like phlogiston that we will later decide are wrong )is a crucial part of the process. Khun talks about this as well, and of course Networking on the Network explains it from the inside.
Naming the element, oxygen, represented a new story about what was happening as things burn. Under the phlogiston theory, people believed that when things burn, they were releasing phlogiston into the surrounding air. Lavoisier instead theorized that when things burn, they absorb a substance from the air, which he named oxygen.
So it's not just the name, it's the story that goes with the name.
Great article, thanks! Almost all of these topics have certainly been swirling around in my head recently too.
"But the choice of which tool to reach for encodes an implicit theory of what embodied intelligence is" -- this is such a great point. The schools of thought that you mention are different in some ways but maybe similar in others. Optimization and RL are not conceptually that different, but where I think they may diverge more is how much you believe there is pre-conceived structure (~model-based) or not (the bitter lesson hypothesis).
About the paradigm, Chris Paxton and I recently wrote (https://www.avikde.me/p/the-first-paradigm-in-robotics-and) about that too from the "VLA+RL" angle as well as hardware commoditization. Looking to computer engineering, it also seemed like the next stages of research would potentially naturally move up the stack or diversify. I'd be curious / interested to see if you think that too, as a continuation of this article.
The whole "moving up the stack" thing is definitely happening. Getting the Spot robot in my lab was a game changer for us. It is making it easier to get more things working better and faster in robotics labs all over the world.
Regarding pre-conceived structure - I think a big mistake people are making right now is black and white thinking about model based vs the bitter lesson. Actually it's a trade-off and there are reasons to be all along the spectrum. I was trying to point to this in our Crossing the Curves post, and also our recent review paper about LLMs and robotics. (https://h2r.cs.brown.edu/wp-content/uploads/cohen24.pdf). There are a lot of reasons to put pre-conceived structure into our systems, and that isn't going to change any time soon (e.g., we aren't (yet) using a learned model for the OS running on our robot). Even within a VLA , we are putting a lot of structure into the data collection, model architecture, and training recipes.
Regarding next stages of research - I think things are going to hit a pretty big wall in terms of programmability and reliability. In my talk at ICRA last week, I quoted Rod Brooks: "Technologies for robots need 10+ years of steady improvement beyond lab demos of the target tasks to mature to low cost and to have their limitations characterized well enough that they can deliver 99.9% of the time. Every 10 more years gets another 9 in reliability." We saw exactly this happen for self-driving cars. Except for humanoids, we're at the beginning of the bubble.
I totally agree with you on the safety / reliability; that and energetic efficiency are my primary concerns with blind pursuit of scaling. While you have a nuanced perspective on the spectrum between scaling and architectural improvements, I'm afraid that the economic incentives (commercially, and maybe to some extent even in academia with grants being funded) push toward the scaling direction for near-term payoff, ignoring the impending wall you pointed out.
I'm very curious about your mention of "programmability". On one hand, VLAs / VLMs and foundation models bring more flexible programming than has been possible before (using prompts). If you have any further reading to suggest on that, I'd love to see it.
While programmability might refer to the high-level code or high-level abstractions we have in language, there is also behavioral programmability, which is what we are all circling in on with how we build these sophisticated physical AI systems. When I simulate a robot or pick up a UMI to generate training data, I am in essence programming the robot, just as much as I would have when tuning gains on my controller to make those policies run on the robot. The difficulty is that there is a delayed feedback loop for the roboticist programming this system, which makes forming the abstraction difficult, and therefore, why we argue this is a pre-paradigm field.
We are definitely in an entirely new way of thinking about high-level abstractions, as you point out, but programming robots (or any mechanical system in the physical world) has always had a translation issue going from language to behavior.
That's a really interesting perspective on the breadth of what "programming" entails with foundation models. In my Ph.D. thesis, I was trying to suggest that programming behaviors = composition of behavioral primitives (https://www.avikde.me/p/phd-defense) so I am very much drawn to non-prompting definitions of programmability. :)
Oh cool you work with Dan? He's the best. Tell him hi. :-) Composition is super important, especially composing heterogeneous primitives (learned, programmed, behavior cloned, etc).
I've been using the term 'Programming the robot" as a way of framing the general HRI question of how we tell the robot what we want it to do (at 5 9s of reliability). We program the robot in many ways:
* active learning (like Tesca Fitzgerald's work, asking questions or specific demos)
* ???
Then the research question is: what is the most efficient way to program the robot to robustly carry out the task(s) you want it to do in the environment you want it to do it. It's probably not any one of the above but some combination. We would all be thrilled if VLMs gave you 95% zero-shot but for almost every application you want 99.99. (Unloading the dishwasher at 95%, dropping 1 in 20 of your dishes, gets old real fast.)
We've also been thinking about tasks where 95% is good enough - like trash pickup.
Besides POMDPs were there any other possible theoretical structure for embodied intelligence that you found most promising?
I always thought of robotics more akin to related engineering fields like early aerospace right now rather than hard sciences. IIRC aerospace really advanced as a science when people could collect data via instruments like wind tunnels and figure out which techniques actually mattered for flight. Perhaps we’re more so in a stage similar to that, where we have no solid framework to test and measure embodied intelligence beyond if it can do tasks we deemed as “hard.”
I'm a pretty big fan of George Konidaris' skills2symbols work (hence why we recruited him to Brown in 2016 ;-)). I also think control theory has some good ideas - observability and controlability, even though I only vaguely understand these concepts.
I don't want to downplay the importance of neural models either - from convolution networks to the transformer to diffpo. At the first CoRL, Leslie Kaelbling said we need six more things like convolution to make intelligent robots.
Really cool article - I loved it! This gives words to something I’ve been thinking about for a while (though in my head I used the example of physics and theories of light before Einstein to serve as an analog to modern robotics).
Two things I find interesting after reading the conclusion:
1. Perhaps we’re waiting for a Lavoisier, but perhaps we also simply don’t have enough evidence/data currently to even approach the right theory (I imagine Lavoisier didn’t come up with the building blocks of the periodic table merely from Priestleys oxygen experiment, but rather from a whole bunch of other empirical evidence; perhaps we don’t have that “critical mass” yet?). Curious what you think about this.
2. It is interesting that we’re making seemingly rapid progress on engineering and very little understanding/science progress. Do you think engineering progress is fundamentally limited by science progress? I’d probably be inclined to say yes to this, but as a counterpoint, it feels like we’ve made relatively little progress on the science of deep learning for instance, but we’ve made very impressive empirical progress in building vision and language systems that really work and do useful things.
Re engineering vs science, I think they both feed off each other, and figuring out Stuff That Works is an essential part of progress. But we can hope for a distillation of the Stuff That Works into something more fundamental and explanatory - that is our missing paradigm.
One of my buddies when I was a postdoc empirically found an approach to RL that worked, and wrote a paper about it. Then, he did some math and derived it more formally. This revealed that his empirical update rule was missing a factor in the denominator. Adding the factor the theory said to add improved performance empirically. It doesn't always work out that way, but that was a beautiful example of the duality at work. Maybe I will go dig up the citations and write a post about it. :-)
I think this idea also connects to your first point - I think it was important that Lavoisier and Priestley and others were doing the same experiments, playing with the same things, and telling stories about what they were observing. And then those stories suggested further empirical things to try. Telling stories, to other people (including the stories like phlogiston that we will later decide are wrong )is a crucial part of the process. Khun talks about this as well, and of course Networking on the Network explains it from the inside.
I dont understand yet, why the problem of giving the element the right name, oxigen, is the same problem we have with naming physical AI.
Naming the element, oxygen, represented a new story about what was happening as things burn. Under the phlogiston theory, people believed that when things burn, they were releasing phlogiston into the surrounding air. Lavoisier instead theorized that when things burn, they absorb a substance from the air, which he named oxygen.
So it's not just the name, it's the story that goes with the name.
Great article, thanks! Almost all of these topics have certainly been swirling around in my head recently too.
"But the choice of which tool to reach for encodes an implicit theory of what embodied intelligence is" -- this is such a great point. The schools of thought that you mention are different in some ways but maybe similar in others. Optimization and RL are not conceptually that different, but where I think they may diverge more is how much you believe there is pre-conceived structure (~model-based) or not (the bitter lesson hypothesis).
About the paradigm, Chris Paxton and I recently wrote (https://www.avikde.me/p/the-first-paradigm-in-robotics-and) about that too from the "VLA+RL" angle as well as hardware commoditization. Looking to computer engineering, it also seemed like the next stages of research would potentially naturally move up the stack or diversify. I'd be curious / interested to see if you think that too, as a continuation of this article.
The whole "moving up the stack" thing is definitely happening. Getting the Spot robot in my lab was a game changer for us. It is making it easier to get more things working better and faster in robotics labs all over the world.
Regarding pre-conceived structure - I think a big mistake people are making right now is black and white thinking about model based vs the bitter lesson. Actually it's a trade-off and there are reasons to be all along the spectrum. I was trying to point to this in our Crossing the Curves post, and also our recent review paper about LLMs and robotics. (https://h2r.cs.brown.edu/wp-content/uploads/cohen24.pdf). There are a lot of reasons to put pre-conceived structure into our systems, and that isn't going to change any time soon (e.g., we aren't (yet) using a learned model for the OS running on our robot). Even within a VLA , we are putting a lot of structure into the data collection, model architecture, and training recipes.
Regarding next stages of research - I think things are going to hit a pretty big wall in terms of programmability and reliability. In my talk at ICRA last week, I quoted Rod Brooks: "Technologies for robots need 10+ years of steady improvement beyond lab demos of the target tasks to mature to low cost and to have their limitations characterized well enough that they can deliver 99.9% of the time. Every 10 more years gets another 9 in reliability." We saw exactly this happen for self-driving cars. Except for humanoids, we're at the beginning of the bubble.
Thanks, I appreciate the thoughtful response!
I totally agree with you on the safety / reliability; that and energetic efficiency are my primary concerns with blind pursuit of scaling. While you have a nuanced perspective on the spectrum between scaling and architectural improvements, I'm afraid that the economic incentives (commercially, and maybe to some extent even in academia with grants being funded) push toward the scaling direction for near-term payoff, ignoring the impending wall you pointed out.
I'm very curious about your mention of "programmability". On one hand, VLAs / VLMs and foundation models bring more flexible programming than has been possible before (using prompts). If you have any further reading to suggest on that, I'd love to see it.
While programmability might refer to the high-level code or high-level abstractions we have in language, there is also behavioral programmability, which is what we are all circling in on with how we build these sophisticated physical AI systems. When I simulate a robot or pick up a UMI to generate training data, I am in essence programming the robot, just as much as I would have when tuning gains on my controller to make those policies run on the robot. The difficulty is that there is a delayed feedback loop for the roboticist programming this system, which makes forming the abstraction difficult, and therefore, why we argue this is a pre-paradigm field.
We are definitely in an entirely new way of thinking about high-level abstractions, as you point out, but programming robots (or any mechanical system in the physical world) has always had a translation issue going from language to behavior.
That's a really interesting perspective on the breadth of what "programming" entails with foundation models. In my Ph.D. thesis, I was trying to suggest that programming behaviors = composition of behavioral primitives (https://www.avikde.me/p/phd-defense) so I am very much drawn to non-prompting definitions of programmability. :)
Oh cool you work with Dan? He's the best. Tell him hi. :-) Composition is super important, especially composing heterogeneous primitives (learned, programmed, behavior cloned, etc).
I've been using the term 'Programming the robot" as a way of framing the general HRI question of how we tell the robot what we want it to do (at 5 9s of reliability). We program the robot in many ways:
* Python
* PLCs
* Natural language
* Demonstrations (LfD, behavior cloning, fine-tuning)
* RLHF
* Sim2real RL
* active learning (like Tesca Fitzgerald's work, asking questions or specific demos)
* ???
Then the research question is: what is the most efficient way to program the robot to robustly carry out the task(s) you want it to do in the environment you want it to do it. It's probably not any one of the above but some combination. We would all be thrilled if VLMs gave you 95% zero-shot but for almost every application you want 99.99. (Unloading the dishwasher at 95%, dropping 1 in 20 of your dishes, gets old real fast.)
We've also been thinking about tasks where 95% is good enough - like trash pickup.
Besides POMDPs were there any other possible theoretical structure for embodied intelligence that you found most promising?
I always thought of robotics more akin to related engineering fields like early aerospace right now rather than hard sciences. IIRC aerospace really advanced as a science when people could collect data via instruments like wind tunnels and figure out which techniques actually mattered for flight. Perhaps we’re more so in a stage similar to that, where we have no solid framework to test and measure embodied intelligence beyond if it can do tasks we deemed as “hard.”
I'm a pretty big fan of George Konidaris' skills2symbols work (hence why we recruited him to Brown in 2016 ;-)). I also think control theory has some good ideas - observability and controlability, even though I only vaguely understand these concepts.
I don't want to downplay the importance of neural models either - from convolution networks to the transformer to diffpo. At the first CoRL, Leslie Kaelbling said we need six more things like convolution to make intelligent robots.