Consider what it would take to recover an animal’s reward function from the outside, with no access to its nervous system and no prior on what it wants. The thought experiment that follows came out of a conversation with Peter Whitney and Meghan Huber about how animals acquire behavior. What is the upper bound on what pure observation of an animal’s behavior can recover in terms of reward?
A wolf living an ordinary life inside a habitat it never learns is instrumented. Every force, sound, and image is recorded, so its behavior can later be inverted for the reward that produced it.
Imagine a mouse in a sealed and fully instrumented enclosure: force plates under every surface, cameras covering every angle, microphones in every corner, and chemical and thermal sensors standing in for smell and temperature. The mouse lives, and dies inside, and the enclosure records every stimulus that reaches it across that lifetime, down to the pressure on each paw and the light entering each eye, so that its sensory history is stored without gaps. Scaling the same construction to a wolf, in a dome a kilometer across, changes the logistics and not the premise, a complete record of the total stimulus each animal receives from birth to death.
The inference runs after the animal is dead and the logs are complete. You saw everything the animal saw and everything it did, and the only unknown is the drive in the middle that turned that input into that behavior. So you ask, in a Bayesian sense, what reward and exploration function would have produced this behavior in this world. You want the posterior over the animal’s reward R given its behavior and the world it lived in: P(R | behavior, world) is proportional to P(behavior | R, world) times P(R). This is inverse reinforcement learning, the problem of recovering a reward from observed behavior, run under an assumption you almost never get to make. Inverse RL is hard for two separate reasons, and the dome removes only one of them. The first is partial observability. Since you never know what the agent actually saw and so you cannot tell whether it behaved strangely because it wanted something strange or because it saw something you missed. The dome removes that problem. Because you hold the complete record of the world, anything the recovered reward still has to explain is a property of the animal. In effect the dome is Laplace’s demon scoped to one animal’s reward instead of every particle in the universe. Quantum mechanics rules out reading the atoms exactly, but the reward is not an atomic quantity, and the behavior that reveals it can be recorded completely.
The second reason survives the dome intact. Inverse RL is under-specified: many different reward functions produce the same behavior, so the data never picks out one of them. Skalse and colleagues showed that many different reward functions produce the same optimal behavior, so even with infinite fully observed data the reward is only partially identifiable. The dome removes partial observability and leaves that non-identifiability untouched. Whatever the inference returns is one member of a large family of rewards that explain the wolf equally well, and picking among them needs assumptions the behavior alone will not supply.
You would expect this to require a faithful physical model of the animal, with accurate muscles, tendons, and contact dynamics, and you would expect a wrong body to produce a wrong reward. Meghan pointed me to work by Dagmar Sternad and Neville Hogan, who spent years measuring what cost humans minimize when they move. The obvious candidate is muscle effort, and their measurements say people do not minimize it. People hold onto solutions that tolerate their own motor noise and sit in a forgiving region of the task, and they will spend extra effort to stay there. The quantity they optimize lives in a coarser space than the mechanics, closer to the structure of the task than to the forces in the limb. If that holds for wolves, kinematic fidelity may be a red herring, and you could recover what the wolf wanted without simulating its body faithfully, because the reward was never stored in the fine detail of the body.
I am thinking about this now because robotics is approaching a solution that is reminiscent of animals. Robots that learn while they run are now conceivable. They will be able to update their weights during deployment rather than only during training, keeping old skills as new ones come in, and utilizing good reward functions to determine what experiences they should go collect next. This same robot is leveraging a similar process that we take for granted in animals under the paradigm.
The cleanest version of that is not a robot but a language model. A language model aligned with reinforcement learning is trained against an objective, whether an explicit reward model or the implicit target of a preference-optimization loss, and after training that objective is latent in the weights the way an animal’s drives are latent in its nervous system. As a practitioner I expect the same phenomena to appear in a system this well described: a reward provably exists, so the question is whether inverse reinforcement learning can recover it. The model is the setting where recovery should be easiest, because the world it acted in is its context window and every token it produced is on disk, so the partial-observability obstacle that makes the animal case hard is absent by construction.
I ran the direct-preference-optimization form of this. For a model aligned under a KL constraint, the reward it was optimized toward is available in closed form as the log-ratio of its token probabilities to those of the reference policy
with no reward model to fit. I computed that log-ratio for the chosen and rejected response of each pair in RewardBench and measured how often it ranks the human-preferred response above the other, against two baselines: the aligned model’s likelihood alone, and the reference model’s likelihood alone.
With the true reference in hand, meaning the specific supervised checkpoint the aligned model was tuned from, the recovered reward ranked human preferences at 0.76, above the aligned model’s own likelihood at 0.75 and the base model’s at 0.71. The objective is reconstructable from the policy. The aggregate hides the structure. On open-ended answer quality, the category the alignment was built to improve, the recovered reward scored 0.95 and approached 1.0 once normalized per token, while the raw likelihood of either model sat near 0.03. On refusals and on grade-school math the ordering inverted, and the recovered reward fell to 0.38 and 0.61 while the base model’s likelihood carried the preference at 0.92 and 0.81. Substituting a public base model for the unavailable true reference removed the effect entirely and dropped the recovered reward below both likelihood baselines.
The reward recovered from the model against the base model’s raw likelihood, across five task types grouped from the 23 RewardBench subsets. The recovered reward matches human preference on the open-ended chatbot answers the alignment was built to improve and falls behind plain likelihood on tricky instructions, grade-school math, and refusing harmful requests. Zephyr-7B with its true reference model; each number is how often the score picks the answer humans preferred.
So even in the system built to carry a clean reward, what came back was narrow and unstable. The reconstruction recovered the specific thing the model was trained to want and missed the axes the training never touched, and whether the raw log-ratio or its per-token normalization counted as the reward changed from one category to the next. The reward was recoverable, and it did not reduce to a single quantity that held across the model’s behavior.
There is an obvious answer to where the wolf’s reward came from, which is evolution. Survival and reproduction are a reward function of a kind. But that signal is close to useless as something to learn from. Evolutionary signals exist purely in the aggregate of whether wolves continue to breed and produce offspring. What evolution can do instead is install proxies that pay out on the timescale the animal actually lives on. Dopamine in the nucleus accumbens carries a reward prediction error, so an animal is scored not on outcomes but on outcomes relative to what it expected. Singh and colleagues made the same argument formally, treating an animal’s reward function as something evolution searches over precisely because fitness itself is not learnable from within a single life. If that is the structure, the dome never recovers the wolf’s fitness. It recovers the accumulated pile of shaping terms that evolution left in the wolf to stand in for fitness.
If the machine works and hands you the wolf’s reward function, I do not know what it looks like. Maybe it factors into quantities you can name, like seeking warmth or avoiding pain or staying near kin. I doubt it. My guess is that the recovered object is high dimensional and tangled, and that no small set of human words covers it, though with enough study we might form abstractions over it the way we pulled temperature out of the motion of particles. What draws me to the problem is what such an object would let you do. Most of robotics is spent modeling one complex behavior at a time, and a recovered reward sits underneath the behavior, closer to the operating system a life runs on than to any single thing it does. If you could extract that and install it, you would be seeding a machine with the objective that organizes a life, and the question I actually care about is whether a machine seeded that way would self-organize into something recognizably like the animal it came from. I cannot build the dome. I think the reward is in there.
References
Sternad, D., Huber, M.E., & Kuznetsov, N. (2014). Acquisition of novel and complex motor skills: stable solutions where intrinsic noise matters less. Advances in Experimental Medicine and Biology, 826:101-124.
Koeppen, R., Huber, M.E., Sternad, D., & Hogan, N. (2017). Controlling physical interactions: Humans do not minimize muscle effort. Dynamic Systems and Control Conference (DSCC).
Zhang, Z., Guo, D., Huber, M.E., Park, S.-W., & Sternad, D. (2018). Exploiting geometry of solution space to reduce sensitivity to neuromotor noise. PLoS Computational Biology, 14(2): e1006013.
Ng, A.Y., & Russell, S. (2000). Algorithms for inverse reinforcement learning. Proceedings of the 17th International Conference on Machine Learning (ICML).
Ziebart, B.D., Maas, A., Bagnell, J.A., & Dey, A.K. (2008). Maximum entropy inverse reinforcement learning. Proceedings of AAAI.
Skalse, J., Farrugia-Roberts, M., Russell, S., Abate, A., & Gleave, A. (2023). Invariance in policy optimisation and partial identifiability in reward learning. Proceedings of the 40th International Conference on Machine Learning (ICML).
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C.D., & Finn, C. (2023). Direct preference optimization: your language model is secretly a reward model. Advances in Neural Information Processing Systems (NeurIPS).
Lambert, N., et al. (2024). RewardBench: evaluating reward models for language modeling. arXiv:2403.13787.