Radiate Love: a 2025 Retrospective
Happy Groundhog Day!
Happy Groundhog Day! David Watkins suggested that I write a Year in Review post, and I guess better late than never. My New Year’s Resolution for 2025 was “Radiate Love.” I tried to be like a lighthouse, radiating love to everyone around me. It’s my job to radiate the love: I’m not responsible for how it’s received, just for sending it out into the world.
My Dad died in August of 2024, and for the first Father’s Day without him, in 2025, I decided it was time to welcome a cat into our family. My Dad always had a Persian cat when I was a kid. After I moved out and went to college, my parents got a Persian with points (Himalayan) named Gizmo. I came to see her every chance I got, but I wondered if she loved me back, or even remembered me. So I decided to teach her tricks. If she learned the trick on one trip home and still remembered it the next trip, then I knew she remembered at least something about me. To my delight, it worked!
For clicker training a cat or a dog, we don’t use a fixed reward function. To teach Gizmo to roll over, I first taught her to lie down, then roll on her side, and then roll all the way over, step by step. After she learned to lie down, I stopped rewarding the lie down, but only when she rolled over. My reward was not fixed but rather depended on Gizmo’s current policy: once I knew she knew how to lie down, I changed my reward function. MacGlashan et al. pointed out that this method of rewarding is not Markov. Instead, humans give policy-dependent feedback that corresponds to the Advantage function: how much better (or worse) an action is relative to the current policy. They showed that an algorithm that takes this difference into account is able to learn more effectively from human feedback.
When training a cat or a dog, it’s also important to keep them focused on you. In fact, distraction training (making sure they listen even in busy, loud environments) is a key step for training service dogs or police horses. When we started this blog, Scott Alexander referenced Rod Brook’s prediction in 2018 that we wouldn’t get an AI that “seems as intelligent, as attentive, and as faithful as a dog” until 2048 But actually, an amazing thing about a dog is their ability to dynamically change the focus of attention as the environment changes. Rod actually specified not that an AI should be attentive, but specifically a robot - a physically grounded embodied agent. And we are not there yet. Our RL algorithms myopically pay attention to maximizing the reward. Neural transformer models apply the idea of attention to a context buffer, but we are still working on extending these models to a multi-scale 3D spatial model of attention on a robot. This ability, to move and point the camera, is a critical missing piece to making a robot that is as attentive and faithful as a dog (or a cat!). It involves the ability to search for and find objects, to have a goal, and to have a state of known information and unknown information that unfolds at a high frame rate over space and time.
My lab is working on one aspect of this problem: resolving human pointing gestures, so that a robot can change its focus of attention in response to a person. Led by Daphna Buschbaum and her student Madeline Pelgrim, we studied how dogs and human toddlers interpret pointing gestures. My Ph.D. student, Ivy He, modeled this behavior in a robot to enable a quadruped robot to follow a point to retrieve an object. But much is still missing - a huge part of the interaction between humans and dogs or toddlers is attention giving and getting. Our next step is to install a 7-degree microphone array on Spot and work on generative models to predict camera movement in response to video and audio input.
Exploring all this with a new kitten has been fascinating. I taught Gummi Bear to lie down, spin, and jump through a hoop. Like with Gizmo, I am using a non-Markov reward function to gradually shape her behavior. And I love her more than I ever thought possible: I did not realize it was possible to have this close of a relationship with an animal, despite growing up with cats and learning to ride horses as an adult. (Future post coming about the the Human–Agent Teaming and Horse–Human Partnerships workshop in Arizona!)
For 2026, my resolution is to “Compartmentalize.” I am an embodied goal-directed creature with many goals in different parts of my life and at different time scales. Compartmentalizing helps me show up in each of these areas, at each of these timescales, without devolving into my Achilles heel: anxiety expressed as worry and rumination. For me, this blog is itself an important part of this resolution, because it is an opportunity to reflect on the connections between my professional life, my hobbies, and my family.
What do you think? Post in the comments a story about a connection between you and an animal and what this tells us about robotics.