A language model starts life as a compression of the internet. It is trained to predict the next token across everything people have written. The next two stages are where it learns to be useful. First came reinforcement learning from human feedback, where people rank outputs and a reward model learns to imitate those rankings. More recently the field moved to verifiable rewards, where the reward is not a learned approximation of human taste but a function that checks an answer against ground truth. A math problem has a known solution, so you can grade the model’s work by comparing to it. A program has unit tests, so you can grade the model’s code by running them. The reward is a small, inexpensive, deterministic function.
The answer key is not the essential part, though. The checker is. OpenAI recently published ten results in mathematics and theoretical computer science, produced by an internal version of the model it has since shipped as GPT-6 Astra, including an explicit non-sofic group and a disproof of Connes’s rigidity conjecture, along with Lean 4 formalizations of each one that anyone can build and check. OpenAI put the compute for the successful runs at roughly $2,000 at API rates, not counting the failed attempts. A proof assistant accepts or rejects an argument without being told what the argument should say, so these problems were far cheaper to evaluate than to solve. Where a checker exists, scoring is cheap whether or not the solution is already known.
Games were the first version of this. Chess and Go come with rules that decide a winner, so self-play generates its own supervision. AlphaZero trained that way without a single human label. Protein structure prediction is the same shape with a different origin. AlphaFold’s reward is the distance between a predicted and measured structure, dense down to the atom and cheap to compute. The measured structures were the expensive part: decades of crystallography and cryo-EM deposited in the Protein Data Bank. The checker was cheap only because somebody else had already paid for the ground truth, offline, once.
When the reward is a function you can call, evaluation is cheap. You can generate a candidate solution, score it, and throw it away millions of times a second, limited only by how fast the hardware runs the forward pass and the checker. Training against a verifiable reward is mostly a matter of buying more compute.
Robotics has no function like that, at least not in general. For narrow, well-modeled tasks the field built accurate analytic models decades ago. Rizzi and Koditschek’s juggling controller kept a ping-pong ball bouncing on a paddle using a “mirror law” derived from the ball’s dynamics. Mason worked out the mechanics of pushing an object across a surface. Howe and Allen built models of grasping and sliding contact. These hold precisely inside the settings they were written for. What we have lacked is a way to turn them into supervisory signals that improve behavior more broadly. Outside the settings they cover, the reward for a manipulation task is not a string you compare against an answer key. It is a fact about the physical world, and the only way to read that fact is to put a robot in the world and let it act. Whether the block ended up stacked, whether the cloth got folded, whether the connector seated, none of these can be evaluated without a rollout on hardware. Each rollout takes real time, wears real actuators, and needs a real person nearby to reset the scene. Where the language model is bottlenecked by compute you can buy, the robot is bottlenecked by robots you have to build, power, and babysit.
Simulation is one way out. If you can model the task in a physics engine, the reward becomes cheap again, because now the world is also a function you can call. But it comes with a cost. The reward you get to write in simulation is usually sparse: did the task succeed, yes or no. A sparse reward gives the policy almost nothing to climb, because most of the time the answer is no and there is no gradient pointing toward yes. To get through that, you need either a policy strong enough to stumble onto success by exploration, or a hand-shaped dense reward that leaks your own assumptions into the behavior, or a curriculum that walks the problem in from something easier. All three are labor, and none of them transfers cleanly across tasks. Simulation trades the cost of the reward for the cost of dealing with sparsity. And even then you have to transfer the policy back to the real world, which is never exactly like the simulator.
There is one class of task where simulation already gets past sparsity. Legged locomotion is the success story of reinforcement learning in sim, because the field found a way to make the reward both cheap and dense. You do not reward a walking policy for the sparse fact that it stayed upright. You reward it, at every timestep, for how closely its motion tracks a reference, so the single success-or-failure at the end of an episode becomes a gradient available on every frame. We get cheap supervision from the reference produced once, offline. It can be motion capture of a real gait, the imitation-reward recipe Peng and colleagues made standard with DeepMimic and that Escontrela and colleagues pushed further by learning a style reward straight from a motion dataset in place of a hand-designed objective, or a dynamically feasible trajectory from a model-based optimizer. On humanoids the recipe is already routine: Cheng and colleagues’ ExBody trains a Unitree H1 to track human motion-capture clips across many millions of simulated rollouts and runs the resulting policy on the physical robot. Those rollouts are affordable because each one is graded against the reference for almost nothing. The motion-capture data is the supervisory signal, and the simulator applies it cheaply at every timestep.
Left: several dozen Unitree G1 humanoids walking forward under a trained velocity-tracking policy in Isaac Lab; the aligned arrows are the velocity command each one is tracking. Right: one G1 tracking a motion-capture clip in MuJoCo, with the reference it is scored against drawn as a red point cloud, the target joint positions for the current pose and the upcoming keyframes it is conditioned on. The dense per-timestep tracking reward is what makes locomotion learnable in simulation; contact-rich manipulation has no equally cheap reference to track. Rendered by us on an RTX 3090 Ti: left in NVIDIA Isaac Sim, right via NVIDIA ProtoMotions in MuJoCo; motion from the BONES-SEED dataset retargeted to the G1.
This is the test a vision-language-action model has to pass. The promise of VLAs is that a big pretrained policy carries enough prior about the world to make exploration tractable, so that success is not vanishingly rare and the sparse reward becomes something a policy can climb. In that framing the interesting quantity is a threshold. On one side, the prior is strong enough that the policy finds success often enough to learn from it, and reward, even sparse reward, is enough to improve. On the other side, success is so rare that the reward never fires and learning stalls. Progress in embodied AI is largely a question of how fast we cross from the second regime to the first, which is the same as asking how cheap we can make a verifiable reward in the real world.
The first two weeks of GPT-6 Astra showed that threshold in public. Within days of the September release, people on X had the model driving robot arms through a harness of camera frames, joint state, and a short list of bounded motion commands, with no robot-specific training. Robocurve ran the cleanest version: a pair of I2RT YAM arms, two tasks, twenty trials per model per task, each trial scored by a human grader against a five-stage rubric. Astra put a red block into a bowl in 19 of 20 trials. Asked to seat a round puzzle piece into its matching groove, it managed 2 of 20, the same count as Claude Fable 5.1. A stronger prior carried the task that was already near the threshold and did nothing for the task that requires fitting a shape into a slot. Every one of those 120 trials was scored by a person watching, who knew which model was running. Between the release and the report, the policy got a generation better. The cost of reading the reward did not move.
Cheap-to-evaluate rewards do exist in the physical world, just not where most manipulation demos are. Consider data center maintenance. A great deal of the work is checking and correcting the state of physical connections: is this ethernet cable seated in this switch port, and if not, seat it. The reward function for that task is almost free to evaluate, because the switch already tells you. The port reports link, or the cable tester beeps, or the light turns green. You do not need a human to judge whether the task succeeded, and you do not need to model human preference to define success. The ground truth is sitting in the network’s own telemetry, as clean a verifiable reward as any unit test. The cost of scoring is relative, though, and it can swing wildly across tasks that look similar. Ask instead whether a robot sliced an apple cleanly and nothing in the world reports the answer. The reward is the same kind of physical fact, but reading it means paying a person to look at every attempt. Vision-language models can already label some of this, but they still struggle with fine-grained physical outcomes. Whether the world scores the work for free or you have to pay for the judgment depends on the task, not the robot.
You can also manufacture the signal. When Gu and colleagues trained real robots to open a door with deep reinforcement learning, from scratch and without demonstrations, they did not put a person behind each arm to grade attempts. They bolted an IMU to the door and computed the reward from its quaternion readings, so the door reported its own angle and two arms ran for hours with nobody scoring them. Scaling it means wiring every door with a sensor. The reward is cheap per attempt and expensive per task, which is the inverse of a unit test, where writing the checker is cheap and running it is cheaper.
Driving that setup cost down is infrastructure work, and it has started. Anthropic recently put out a research preview of the Model Hardware Standard, a shared specification that lets an agent discover and drive physical instruments through common read and write primitives instead of a bespoke driver per device. The stated target is scientific automation. Early users at Genentech, Carnegie Mellon, and the University of Washington are using it to coordinate liquid handlers, robot arms, and plate readers. The framing is autonomy, but the same interface is a reward channel. A plate reader that reports its measurement over a standard read is a door with an IMU on it, generalized to an entire instrument catalog. If reading the state of the physical world becomes a call you can make, the cost of wiring up a new task falls from building a custom rig to using a driver that already exists.
The expense in robotics is not evenly distributed. The reward function for the cable can be nearly free. What stays expensive is everything around it: running a real robot in a real aisle, and simulating the task well enough to train the robot before it goes there. Contact between a stiff connector and a socket, the flex of the cable, the friction of the housing, these are the regimes physics engines model worst, so the cheap reward is trapped behind an expensive rollout and a simulator that gets it wrong.
The cost of a reward is not fixed. Locomotion got its cheap signal because someone found the right intermediate representation, a reference trajectory, and the field keeps finding more of them. Rewards that were once intractable to compute keep getting cheaper. Judging whether an image matches a description used to take a person, and a vision-language model now does it in a forward pass. Estimating human pose from a single video, the thing that lets motion capture scale past a lab, was a research problem a decade ago and is close to a library call today. Every time a piece of perception or judgment that used to need a human gets absorbed into a model you can call, a reward that used to be expensive turns back into a function. Two results from the same month point the same way. Akira Sasaki had Astra design a quadruped and train nine gaits for it in simulation with reinforcement learning, 25 iterations over five days, and others had it build an Isaac Sim environment, configure PPO, and tune the run. Eureka did this in 2023 with GPT-4 writing the reward code for Isaac Gym tasks and beating the hand-written rewards on most of them. Separately, R2S-Eval reports that vision-language models judging pairs of manipulation rollouts agree with human annotators 91.9% of the time in scenes calibrated real-to-sim. Neither is a verifiable reward. A pairwise preference is not a success signal, and a model-written reward carries the model’s assumptions the way a hand-shaped one carries yours. Both move work that used to be a person’s into a forward pass. As we get better at building these intermediate signals, more tasks get a cheap reward, and some that look out of reach now will not stay that way.
This is where the effort should go. We spend most of our attention on the policy, on architectures and pretraining and scale, which is the part the rest of machine learning already knows how to push on. The scarcer thing is verifiable rewards in the physical world that are cheap to evaluate and honest about success, and tasks where the world grades itself for free are especially valuable. Verifiable digital rewards became cheap because someone could write the checker. For robotics, the question is which tasks the world already scores on its own.
References
Mason, M.T. (1986). Mechanics and planning of manipulator pushing operations. International Journal of Robotics Research, 5(3).
Howe, R.D., & Cutkosky, M.R. (1996). Practical force-motion models for sliding manipulation. International Journal of Robotics Research, 15(6).
Burridge, R.R., Rizzi, A.A., & Koditschek, D.E. (1999). Sequential composition of dynamically dexterous robot behaviors. International Journal of Robotics Research, 18(6).
Miller, A.T., & Allen, P.K. (2004). GraspIt!: a versatile simulator for robotic grasping. IEEE Robotics & Automation Magazine, 11(4).
Gu, S., Holly, E., Lillicrap, T., & Levine, S. (2017). Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. IEEE International Conference on Robotics and Automation (ICRA). arXiv:1610.00633.
Farshidian, F., Neunert, M., Winkler, A.W., Rey, G., & Buchli, J. (2017). An efficient optimal planning and control framework for quadrupedal locomotion. IEEE International Conference on Robotics and Automation (ICRA). arXiv:1609.09861.
Peng, X.B., Abbeel, P., Levine, S., & van de Panne, M. (2018). DeepMimic: example-guided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics (SIGGRAPH), 37(4).
Silver, D., Hubert, T., Schrittwieser, J., et al. (2018). A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419).
Jumper, J., Evans, R., Pritzel, A., et al. (2021). Highly accurate protein structure prediction with AlphaFold. Nature, 596.
Escontrela, A., Peng, X.B., Yu, W., Zhang, T., Iscen, A., Goldberg, K., & Abbeel, P. (2022). Adversarial motion priors make good substitutes for complex reward functions. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
Cheng, X., Ji, Y., Chen, J., Yang, R., Yang, G., & Wang, X. (2024). Expressive whole-body control for humanoid robots. Robotics: Science and Systems (RSS). arXiv:2402.16796.
Jenelten, F., He, J., Farshidian, F., & Hutter, M. (2024). DTC: Deep tracking control. Science Robotics, 9(86).
Ma, Y.J., Liang, W., Wang, G., Huang, D., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L., & Anandkumar, A. (2024). Eureka: human-level reward design via coding large language models. International Conference on Learning Representations (ICLR). arXiv:2310.12931.
OpenAI (2026). Ten advances in mathematics and theoretical computer science. Lean 4 formalizations at github.com/openai/ten-proofs
Anthropic (2026). Model Hardware Standard: a research preview. anthropic.com/news/model-hardware-standard-research-preview
OpenAI (2026). GPT-6 Astra: a new generation of intelligence. openai.com/index/gpt-6-astra
Robocurve (2026). GPT-6 Astra on robot arms. openai.robocurve.org/gpt-6-astra
R2S-Eval (2026). Robot evaluation with real-to-sim calibration via vision-language models. r2s-eval.github.io
Sasaki, A. (2026). GPT-6 Astra designs and trains a quadruped. Post on X. x.com/gclue_akira/status/2098300921658868185