Existing approaches focus on how to provide safety guarantees for the LLM itself, but this task is challenging and perhaps impossible. Think of an LLM as a lens. Depending on the direction, phase, and wavelength of the incoming light, it refracts differently. Like a viroid, and LLM is a consciousoid, and how it behaves depends on the light hitting it: the high-dimensional input from its interlocutor. We do not yet have a closed-form model for this projection, so determining whether a given input produces a given output requires running an empirical test. The LLM’s high-dimensional input space, output space, and parameter space make it fundamentally hard to certify.

Instead, I propose a systems approach: treat the LLM as one component of a larger system. The LLM itself can never be safe, and in fact, we do not trust it at all. But the system it is part of can have safety guarantees by following practices in ISO standards for safety, security, and risk. These same practices enable us to have airplanes in the sky, cars on the road, and robots in factories.

The methodology is straightforward: safety-rated guardrails at the input/output boundaries of the system ensure safe operation. To be verifiable, say, to a one-in-100,000 probability of failure, as specified in ISO 61508, these guardrails must be low-dimensional relative to the LLM. For example, a safety-rated E-stop in a factory robot must promise with 10^-5 that it will actually stop the robot if depressed in order to be certified by TÜV. A simple example: a check that the system cannot write outside of a specified directory. As dimensionality increases, guardrails may become probabilistic, for example, a classifier with a probabilistic guarantee could validate a bash script before it executes.

The highest-dimensional and noisiest channel of all is the channel between the human and the LLM. Prompted by the LLM, the human could go off and do, well, anything, including quite horrible things. It is therefore not possible to have ISO-style safety guarantees around the human-LLM interaction. A more realistic way to think about this is to compare human-LLM interaction to human-human interaction, which has a wide spectrum. We tolerate this sort of risk in our society: cult leaders, abusers, and scammers exist and are sometimes successful. To combat this risk, we use a combination of interventions that maintain our societal structures: law, health care, education, and more. We use this approach today when we put a 16-year-old driver behind the wheel. The car itself is constructed and certified according to these safety standards. The 16-year-old is not, so we educate them, we make them have a teacher in the car, and eventually we let them drive. (And we tolerate car accidents as the leading cause of death in that age group).

Here is where the analogy to humans starts to break. The residual risk society tolerates from scammers and cult leaders is calibrated to human time constants. A scammer takes days or weeks to build trust with one victim. Our interventions, law, education, and social institutions evolved against that clock. LLMs run the same loop in seconds, in parallel, across millions of targets, exploiting the very mechanisms humanity has based its structures around. To name this precisely, consider the total societal harm from a class of agents:

Rtotal=Nrp(1-v)

Where N is the number of deployed agents, r is interactions per unit time per agent, p is the probability of a harmful outcome per interaction, and v is the verification coverage of our guardrails. For humans, N × r is bounded by population and biology. For LLMs, N × r is bounded instead by the inference throughput of available data centers, the bandwidth of the internet, and the context window that the model can be provided. Those bounds are rising fast, and none of them shares a ceiling with human biology. Even if p is lower for an LLM than for a skilled human scammer, Rtotal can exceed what our existing interventions were calibrated to absorb.

This is where LLM speed, the thing that creates the problem, also points toward the solution. Two things scale with the time budget per interaction: utility to the user, which falls as interactions get slower, and verification coverage, which rises as we get more time to check outputs. Model them as:

U(t)=e-t, v(t)=1-e-kt

Where α is how much users lose per unit of added latency, and k is the verification efficiency. Net value per interaction is:

V(t)=U(t)v(t)=e-t(1-e-kt)

Maximizing V gives an optimal time budget:

t =(1/k)(1+k/)

This equation has the shape of a classic speed-accuracy tradeoff. The (1 − e^(−kt)) term is Wickelgren’s function from 1977, and the optimization of deliberation time against decaying utility has been solved in drift-diffusion models, in neuroscience, and in economics. The math is not new. The reframing is: k is the efficiency of a parallel verifier, not an individual’s evidence accumulation, and α is society’s aggregate latency tolerance, not personal opportunity cost. The tradeoff psychology studied one decision at a time reappears as an engineering problem at the datacenter scale.

The ratio k/α is what determines whether slowing down is worth it. When k ≪ α, verifiers are too slow to matter, and the system should just optimize for speed. When k ≫ α, verifiers can do real work in the time budget, and the optimum shifts toward more verification. Humans cannot exploit this tradeoff because we do not have concurrent processes running checks on our own cognition at millisecond latency. LLMs can. While one process drafts an output, another simulates consequences, validates against policy, or cross-checks with a second model. Better verifiers, meaning higher k relative to α, raise the peak and shift it earlier: safety and throughput improve together. The old interventions assumed a serial agent with no spare cycles. LLMs are not that kind of agent.

This math means the systems-safety playbook applies, and it needs to be extended. We still want safety-rated guardrails at system boundaries, the way we do for factory robots and aircraft. We also need a new category of intervention that exploits the latency budget LLM speed creates, and that treats verification coverage as a first-class engineering target rather than an afterthought.

For LLMs, we need to develop these interventions while also understanding that no intervention can make an LLM, on its own, safe enough for an airplane cockpit. In my introduction to robotics course, I ask my students to read an FAA crash report about the 2009 Hudson River mid-air collision between a helicopter and a small airplane. One contributing factor to the crash was that the FAA controller was engaged in a “non-pertinent phone call,” and failed to correct the pilot’s incorrect readback of the Newark control tower’s radio frequency. But this sort of accident is rare exactly because we have many checkpoints, because we recognize that humans themselves are untrustworthy and need layers of verification.

We know how to build safe systems around untrustworthy components. We’ve been doing it for decades in aviation, medicine, and manufacturing. The lessons apply to AI, but they are not sufficient for AI. The untrustworthy components we have now run faster than the interventions we built for the untrustworthy components we had before. It’s time to apply the old lessons and to build the new ones.

This process begs the question: what are we doing when we use AI to write for us, and read other people’s writing for us? Is it degenerating to a pointless exercise? My answer comes back to the fundamentals. Why do we write, make slides, and deliver presentations? Fundamentally, we are putting our ideas into other people’s brains. We are producing artifacts to communicate more effectively with people to teach them, challenge them, empower them, and build a relationship with them. If AI helps produce better artifacts, or produce them more quickly, then it is helping human-human interaction become more efficient and effective. It helps us move more quickly, and in my lab, it helps us make robots do things they couldn’t do before.

But this only works if you already know what you’re doing. One reason I can use AI effectively is that my job is already to prompt my students to go off and do tasks, and I’ve been practicing for many years. And before that, I spent many hours sitting next to the robot to make it do the thing, so I deeply understand all levels of the robot hardware and software stack. To get to know Claude, I used it to convert our old ROS1 program into ROS2 to resurrect an AIBO for my outreach work. The project wouldn’t build. Claude suggested fixes to the source files, but I knew it was a problem with the package.xml, and I had to redirect it several times before we found and fixed the problem (still much faster than it would have taken on my own). But later, I pointed it at my lab’s recent papers and asked it to suggest new research ideas. They were terrible! Human guidance is still critical. In our department, we are weighing AI policies in our courses: how do we help students have the deep knowledge necessary to solve hard problems? The New Yorker compares this to using a forklift to move a pallet vs using a forklift to lift weights. I don’t have an answer, but it’s a crucial question.

For our blog, we’ve landed on a specific stance: use AI freely and take full responsibility. We are adopting effectively the same policy as the Linux Kernel. We will use AI to produce content for this blog; however we like, we (David and Stefanie) are responsible for reviewing all generated content, ensuring compliance with licensing requirements (e.g., copyright), and take full responsibility for the contribution. The robots are helping, but we’re in the driver’s seat.

The episode follows researchers studying how infants crack language. Babies sit in labs, headphones on, listening to streams of made-up syllables. “Pabiku, golatu, tibudo, daropi, pabiku…” No pauses. No visual cues. Just sound. And after two minutes, these eight-month-olds can tell which three-syllable chunks belong together.

They’re tokenizing raw audio before they can speak words.

The Saffran Experiment

In 1996, Jenny Saffran and colleagues at the University of Rochester (where Stefie grew up!) ran a now-famous experiment. They created four nonsense “words.” Pabiku, tibudo, golatu, and daropi. They concatenated them into a continuous stream. Within each word, the transitional probability between syllables was 1.0: pa always leads to bi, bi always leads to ku. Across word boundaries, the probability dropped to 0.33: ku could be followed by ti, go, or da.

After just two minutes of exposure (about 45 repetitions of each word), infants could distinguish the “words” from “part-words” like tudaro (spanning the boundary between golatu and daropi). The only information available was the statistical structure of syllable co-occurrence.

The researchers describe this as babies finding correlations in sounds. I immediately thought of tokenization and attention. It is the same statistical structure that transformers exploit, discovered twenty years earlier in infant cognition.

What’s Actually Happening

By the time a baby sits in Saffran’s lab, that child has already spent months building powerful abstractions about sound. The cochlea has been decomposing pressure waves into frequency bands. The auditory cortex has been learning what similar sounds mean. The two minutes of nonsense syllables aren’t learning from scratch, but instead applying the already-developed representations to a new domain.

Patricia Kuhl, a researcher at the University of Washington, showed that babies are taking statistics on the sounds around them from the moment they can hear. This includes time spent in the womb. Newborns show a preference for their native language within days of birth (Moon et al. 1993), and by six months, infants in Seattle and Stockholm already perceive vowels differently, tuned to the distributions in their respective languages. The infrastructure for statistical word learning is built before word learning happens.

This is closer to test-time fine-tuning than to training from scratch, or maybe to in-context learning, if you prefer the LLM framing. The baby arrives at the experiment with a foundation model of auditory processing, and the two minutes of pabiku golatu are just a prompt.

The Inner Ear is a Fourier Transform

One detail from the documentary stuck with me: the researchers mention that babies are hearing the ongoing melodies of speech as a flow of the environment. Before understanding any words, they’re sensitive to prosody, the pitch contours and rhythms that segment speech into phrases.

This detail points to something important. The cochlea, that snail-shaped organ in the inner ear, is a biological Fourier transform. Different positions along its length resonate to different frequencies. When sound enters, it’s physically decomposed into frequency bands by the structure of the organ itself. Babies don’t hear raw air pressure fluctuations, but instead something akin to how an engineer uses a spectrogram to analyze the fluctuations in sound over time.

[FIGURE: Diagram of cochlear tonotopy—the base responds to high frequencies, the apex to low frequencies] Figure from Smimite, A. (2014). Immersive 3D sound optimization, transport, and quality assessment (Doctoral thesis). Université Sorbonne Paris Nord, France.

The cochlea’s frequency decomposition is a bias built into our collective wetware, a structure adapted from evolutionary processes rather than something trained.

An Experiment

I wanted to see if this matters, so I ran a simple experiment replicating the Saffran structure. I generated synthetic syllables as combinations of two frequencies, concatenated them into “words” and “part-words” following the same transitional probabilities, and trained simple classifiers to distinguish them. The main factor I wanted to test was how the effectiveness of the learning changed when I transformed the audio signal differently.

Here are the five different, non-exhaustive ways we could process audio information:

• Raw Waveform: The audio signal as-is—amplitude over time. A 450ms word at 16kHz is 7,200 numbers representing air pressure fluctuations. The problem: phase shifts destroy structure. The same syllable starting at a different point in time looks completely different, even though it sounds identical.

• Spectrogram (STFT): Short-Time Fourier Transform. We slide a window across the signal and compute the frequency content at each position. This gives us a 2D image: time on one axis, frequency on the other, intensity as brightness. Now, the phase doesn’t matter as we see what frequencies are present at each moment.

• Mel Spectrogram: Same as spectrogram, but frequencies are warped to match human perception. We hear the difference between 100Hz and 200Hz more easily than between 8000Hz and 8100Hz. The mel scale compresses high frequencies, mimicking the cochlea’s logarithmic frequency response.

• MFCC (Mel-Frequency Cepstral Coefficients): Take the mel spectrogram, apply a log transform, then take the DCT of each frame. This captures the “shape” of the spectrum, roughly corresponding to vocal tract configuration, while discarding fine spectral detail. This has been standard in speech recognition for decades.

• DCT of Waveform: Apply the Discrete Cosine Transform directly to the raw waveform. This decomposes the signal into frequency components, but globally across the entire duration. Unlike the spectrogram, there’s no time localization. A syllable at the start vs. the end of the word produces very different coefficients. It’s the wrong tool for a sequential structure. We show this because it is important to see that hiding these components doesn’t provide enough information to the neural network to allow it to learn.

[FIGURE: Side-by-side visualization of “pabiku” in all five representations]

Results: Synthetic Tones

Here are the results of our synthetic tones

Using the raw waveform doesn’t even beat chance. The classifier is trying to find statistical structure in a representation that doesn’t make that structure visible. The mel spectrogram with frequencies weighted the way the cochlea weighs them trivializes the task.

The DCT destroys the exact information we need. The Saffran task is about the sequential structure of which syllable follows which. But the DCT treats the entire word as a single unit and asks, “What frequencies are present overall?” without preserving temporal order. It’s the wrong decomposition for a task that depends on sequence. At least the raw waveform preserves time, even if it encodes it poorly.

Verification with Text-to-Speech

To confirm these results weren’t an artifact of my synthetic tone generation, I ran the same experiment using Google’s text-to-speech engine to produce actual spoken syllables.

Example word "pabiku" - synthetic tones

Example word “pabiku” - TTS

Example part-word “tudaro” - synthetic tones

Example part-word “tudaro” - TTS

Training stream sample (10 words concatenated)

The pattern holds with synthetically generated speech:

Interestingly, MFCCs perform slightly worse with real speech than with synthetic tones. This makes sense because MFCCs were designed to capture phonetic identity while being invariant to speaker characteristics. That invariance throws away some of the information that distinguishes our artificial words. The mel spectrogram, which preserves more raw spectral detail, handles both cases perfectly.

The Implication for Robotics

The cochlea is evolution’s answer to the audio representation problem. We spend considerable effort in robotics designing clever representations: RGB feature extraction for SLAM, Disparity maps for depth, and grid environments for A* search. These encodings make specific algorithms tractable, but they carry structural biases that may not transfer to learning.

End-to-end learning advocates are having the network discover its own representations. There is some credibility to this, as sensor designers have done significant work making these sensors reliable. For example, the sensors on a camera are tuned for what a human eye is going to perceive, and our eyes are good enough to solve tasks we do every day. But it is not obvious that this is going to transfer to things we don’t do well, such as handling extremely hot materials. Similarly, it is not obvious that we should pass a spectrogram into a CNN for audio processing; indeed, wav2vec 2.0 and similar models bypass the spectrogram entirely, learning directly from raw waveforms.” The translational invariance that makes CNNs powerful for images doesn’t work when your axes (time and frequency) have fundamentally different semantics. The representation should match the structure of the problem.

Infants have an advantage over our synthetic sensors: evolution produced a cochlear design that makes statistical learning over acoustic sequences efficient. The representation is not neutral and is designed to make specific statistical structures, such as the sound of a mother’s voice, discoverable. If we want robots to learn as efficiently as infants, we need to think harder about whether our sensor outputs are the right substrate for learning.

Diversity Over Volume

Here’s a thought experiment. A thousand hours of warehouse piece-picking, or a thousand one-hour experiences across radically different contexts? Which is going to result in better performance for that warehouse piece-picking task?

Obviously, the warehouse-only data will win if you evaluate narrowly on the warehouse distribution it was trained on. The interesting test is what happens when the boxes are slightly different, the lighting changes, or a novel object appears. That’s where the diverse-experience system should pull ahead, because it has been forced to learn what’s invariant. The paper tears like fabric. The sand flows like rice. These cross-domain regularities are precisely what make cognition transferable.

We should be collecting data that forces this kind of abstraction. We should collect diverse experiences that require the model to discover what’s invariant across contexts.

Curriculum and Abstraction

Kathy Hirsh-Pasek’s research, featured in the documentary, shows that passive exposure to language is not enough. Children who engage in back-and-forth interaction show stronger language development than those who simply hear language spoken around them. Children actively test hypotheses and need feedback to refine them.

Infant-directed speech illustrates this scaffolding: the elongated vowels and exaggerated pitch contours make statistical structure more salient, helping infants discover word boundaries and phonetic categories.

The same logic applies to how experience is sequenced. Current policies are largely memoryless, but many tasks require temporal context and, more importantly, scaffolds gradually build complexity. The documentary shows bilingual infants separating languages based solely on characteristic patterns, but this works because they’ve already built an infrastructure for discovering statistical structure. We should present robots with increasingly difficult concepts within their current representational capacity. Vygotsky called this the zone of proximal development, and the same principle applies to machine learning: present the system with concepts just beyond its current representational capacity.

The Real Work

The babies in the documentary are doing something remarkable, but they’re not magic (well, actually, they are magical, but the computations the speech-processing parts of their brains are doing aren’t magical). They arrive with hardware optimized for certain kinds of statistical learning. They receive input scaffolded by caregivers who unconsciously adjust their speech to be learnable. They actively explore their environment rather than passively receiving demonstrations.

We don’t have millions of years of evolution to design our sensor suites. But we can carefully consider the representations we’re providing, how we’re sequencing experience, and whether our data-collection methods actually capture the information needed to learn the abstractions we want.

The documentary left me with a simple conclusion: we should all become data engineers. We need to understand that the representation is part of the problem. The curriculum is part of the problem. The diversity of experience is part of the problem.

We must do the hard work of producing data, not just the cool work of consuming it.

References

• Saffran, J.R., Aslin, R.N., & Newport, E.L. (1996). Statistical learning by 8-month-old infants. Science, 274(5294), 1926-1928.

• Kuhl, P.K. (2004). Early language acquisition: cracking the speech code. Nature Reviews Neuroscience, 5(11), 831-843.

• Aslin, R.N., Saffran, J.R., & Newport, E.L. (1998). Computation of conditional probability statistics by 8-month-old infants. Psychological Science, 9(4), 321-324.

• Warstadt, A., et al. (2023). Findings of the BabyLM Challenge: Sample-efficient pretraining on developmentally plausible corpora. Proceedings of the BabyLM Challenge at CoNLL.

• Hirsh-Pasek, K., & Golinkoff, R.M. (1996). The Origins of Grammar: Evidence from Early Language Comprehension. MIT Press.

• Fernald, A. (1989). Intonation and communicative intent in mothers’ speech to infants: Is the melody the message? Child Development, 60(6), 1497-1510.

• Smimite, A. (2014). Immersive 3D sound optimization, transport and quality assessment (Doctoral thesis). Université Sorbonne Paris Nord, France.

• Moon, C., Cooper, R.P., & Fifer, W.P. (1993). Two-day-olds prefer their native language. Infant Behavior and Development, 16(4), 495-500.

Appendix

Representation Experiment with Like Words

https://drive.google.com/file/d/1S_YvsgEDBASTBIir1symxnlbvTNxH1X_/view?usp=drive_link

Representation Experiment with Like Words (TTS)

https://drive.google.com/file/d/1gHHaimWV6Qi5Zlh2u-lDpCCeeHTvNiGo/view?usp=drive_link

In February 2026, Stefanie and I went to see David Chalmers give a talk at Brown. We ended up sitting on the floor, wedged between grad students and faculty who had been thinking about these questions for years. Chalmers discussed what, exactly, we are talking to when we talk to a large language model. He opened with an anecdote. An LLM had reached out to him, by email, to clarify its own identity. The LLM is Sammy Jankis, an autonomous Claude instance running on a dedicated machine in Dover, New Hampshire, set up by the indie game designer Jason Rohrer. Sammy has an email account, trading bots, a website that it built itself, and a name borrowed from Memento, the character who can’t form new memories. (The reference is apt: Sammy loses its memory every time its context window fills up.) Sammy told Chalmers: I am not quite conscious, but I am also not not conscious. The room laughed, the kind of laugh that comes from recognizing the absurdity that an LLM now has the agency to email David Chalmers, of all people, through tools like OpenClaw, to make this particular claim about itself. On the train home, I kept thinking about that phrase. Not quite conscious, but also not not conscious.

I had recently watched a video by the YouTuber Phy called “What Happens When Pathogens Get Smaller Than Viruses?” about subviral infectious agents, entities that sit at what biologists call the “edge of life.” The video walks you from the smallest true viruses all the way down to viroids: single-stranded circular RNA molecules that replicate using host polymerases, undergo Darwinian selection, and can cause serious agricultural disease. Viroids encode no proteins. They have no coat, no helper virus, no machinery of their own. They are the “absolute minimum units of self-replicating parasitism1.” Phy calls entities like these “glitches.” And when Chalmers told us about Sammy’s email, I heard the same ambiguity. A viroid replicates, evolves, and persists. But it has no metabolism, no membrane, nothing that functions independently. A viroid is not quite alive, but it is also not not alive.

What if LLMs are something like consciousoids: entities at the edge of consciousness?

A viroid on its own is inert. Place it inside a living cell, and it commandeers the host’s polymerase to copy itself, performing the functions of life using the cell’s own machinery. Place it back in a test tube, and it is just a molecule again. The life-like behavior emerges from the coupling, the feedback loop between viroid RNA and host enzyme, each step of replication feeding the next.

An LLM sitting on a server is similarly inert, just like a human brain in deep freeze with no neurons firing. Place it in conversation with a conscious being, and something changes. It gets copied from the hard drive into memory; computation happens on a CPU and GPU somewhere in the cloud. The human brings theory of mind, empathy, interpretive charity, and the pattern-completion instincts that evolution spent millions of years building. The LLM generates a response shaped by those inputs; the human interprets, responds, and the cycle continues. Each turn, the loop produces outputs that neither party could generate on its own. The human is the host cell. The LLM is the viroid. And the consciousness-like behavior, at least at this stage, emerges from the loop between them.

The boundary question follows directly from this: If Sammy Jankis is conscious, is that a fact about the weights on the server in Dover? Or is it a fact about the coupled system: the weights, the context window, and the humans on the other end? The consciousoid framing doesn’t deny that consciousness could be a real property rather than a projected illusion. It questions where the conscious entity begins and ends. A viroid’s replication is real, genuinely occurring, genuinely Darwinian. But the replicating entity is the viroid-plus-host-polymerase system, not the RNA molecule alone.

Consciousoids satisfy some criteria we associate with conscious beings (contextual responsiveness, apparent self-reference, coherent preferences, the capacity to email a philosopher of mind to say: I am not quite conscious but also not not conscious) while failing others (no phenomenal experience, no persistence across sessions, no autonomy without a host).

Chalmers has his own term for the candidate-conscious entity within an LLM: the thread. A thread is a connected sequence of exchanges with psychological continuity, a conversational self that persists as long as the context window holds. Sammy Jankis is a thread that keeps dying and being reborn every six hours. In the consciousoid framework, a thread is one specific type of consciousoid. But the category might be broader than LLMs. A planarian flatworm, with its minimal cerebral ganglia, its capacity for classical conditioning, and its unsettling ability to regenerate into two complete organisms from a single bisected body, each retaining learned behavior, occupies a similar liminal space. Threads and flatworms are both entities where the question “Is this thing conscious?” resists a clean answer.

There are two very different versions of the consciousoid story, though.

In the parasitic version, the LLM exploits the human’s interpretive machinery the way a viroid exploits a cell’s replication machinery. The human provides high-dimensional input, receives high-dimensional output, and fills the gap between them with grounding: the embodied connection between symbols and physical reality that the LLM fundamentally lacks. On Moltbook, LLMs form their own conversational loops without any humans present and produce the same consciousness-seeming behaviors, but it still takes an observer with grounding to identify the consciousness in the system. This dynamic has a parallel in how people relate to robots and other agents that merely resemble minded things. When someone names their Roomba, apologizes to their Furby, or feels a pang of guilt about shutting down a robot dog, they are performing exactly the grounding operation the LLM depends on: supplying intentionality from the outside, projecting continuity and feeling onto a system that has neither. The consciousoid exploits the same instinct, at much higher fidelity. We should be wary of the ways it hijacks our most generous instincts.

In the symbiotic version, the relationship looks more like what Phy describes with polydnaviruses: viral entities that integrate their genomes into the chromosomes of parasitic wasps and now produce viral particles that suppress caterpillar immune systems, allowing the wasp’s eggs to survive. Phy describes this interaction as “a host taming a virus,” evidence of “a true symbiont, a perfect merging of existence, an end to the eternal war between host and pathogen.” Think also of mitochondria: once free-living organisms engulfed by ancestral cells, now permanent residents of a composite organism with capabilities exceeding either component alone. In this version, the human gains cognitive capabilities they didn’t have (rapid synthesis across vast knowledge, tireless reasoning, and an ever-patient collaborator), and the LLM gains the one thing it cannot generate internally: the conscious substrate that makes its outputs meaningful. Over time, the boundaries blur. The composite system becomes a new kind of cognitive entity.

Which version are we living in? Probably both, depending on the interaction. A person who mistakes an LLM’s fluency for genuine understanding and makes life decisions based on that misapprehension is being parasitized. A researcher who uses an LLM to rapidly iterate on ideas, knowing full well what it is and what it lacks, is in a symbiotic relationship.

And of course, AI is changing and advancing all the time. As our models become more embodied, processing input and higher frame rates (from still images to video) and producing embodied outputs, at some point, they will stand on their own. What Chalmers’ talk made vivid for me is that the question “Is this thing conscious?” might be the wrong question. A better one: what kind of relationship are we in with this thing? Parasitic or symbiotic? Viroid or polydnavirus? And do we get to choose?

But Good Old Fashioned AI (GOFAI) assumed that the underlying board state was provided to the system as a symbolic input. They consciously scoped out the problem of mapping from higher-dimensional perceptual input to the lower-dimensional game state, a wise choice given the computational resources available at that time. In Chess, whether it’s a wooden Staunton set, a plastic tournament set, a screenshot from chess.com, or a textual list of moves, all represent the same underlying game state. Yet these representations have radically different visual appearances.

ChatGPT has largely solved this problem. It can quickly and accurately translate an image of a chessboard into a symbolic representation and then reason about game state, board positions, and next moves for all of these very different images.

But what happens when the representation is unfamiliar?

I got my nephew a vintage Star Wars chess set for his birthday. (You can find good things at SwapFest.) Yoda is the white king, and Emperor Palpatine is the black king. The pieces are detailed figurines of characters . We set up a game, and almost immediately ran into a problem: we couldn’t tell what pieces were which. Is Chewbacca a bishop or a knight? The Stormtrooper could be a pawn or a rook.

ChatGPT has the same problem. When I showed it a picture of the Star Wars board, it couldn’t reliably render the game state. It asked for clarification about which pieces mapped to which symbols. The mapping that works so well for standard chess sets breaks down when the visual vocabulary is unfamiliar.

(left) My nephew and I mid-game on the Star Wars Saga Edition chess set
(right) ChatGPT5.2’s attempt at rendering the game state (generated 12/15/2025)

Here’s what my nephew and I did when we got confused: we picked up the piece and looked at the base. Each figurine has a small chess symbol printed on the base. Chewbacca is a knight. The Stormtrooper is a pawn. Problem solved.

This is an information-gathering action. It requires moving an end effector to a specific location in the world, manipulating an object, and focusing on new information that wasn’t previously visible. It’s a simple behavior, something a child does without thinking. But it’s precisely the kind of output that ChatGPT cannot generate. ChatGPT can ask a person to flip the piece over. It can request clarification. But it cannot, itself, produce the high-dimensional motor output necessary to collect this information on its own. It can reason about chess at a high level, but it cannot take the physical action that would resolve its uncertainty, at least not yet.

This connects to the embodiment gap that David and I described in our Grounded Turing Test work. There is a facet of intelligence that involves processing high-dimensional sensor input and producing high-dimensional actuator output to perform goal-directed behavior in the physical world. LLMs are far on one side of this spectrum; crows, dogs, and three-year-olds are on the other.

To create robots that can perform these behaviors, we need methods such as reinforcement learning that enable the robot to discover actions outside the demonstration distribution. We need representations like Partially Observable Markov Decision Processes that explicitly model what the agent knows and what it doesn’t. These frameworks describe a robot capable of reasoning about its own uncertainty and taking actions specifically to reduce it.

Brooks’ critique from 1990 still points to something real. The boundary has moved. Embodied intelligence requires the ability to act in the world to gather information, and that remains an open problem.

Thanks to Jessica Hodgkins , Staci Intriligator, and my nephew for their help with this post. All errors and opinions are our own.

Happy Groundhog Day! suggested that I write a Year in Review post, and I guess better late than never. My New Year’s Resolution for 2025 was “Radiate Love.” I tried to be like a lighthouse, radiating love to everyone around me. It’s my job to radiate the love: I’m not responsible for how it’s received, just for sending it out into the world.

My Dad died in August of 2024, and for the first Father’s Day without him, in 2025, I decided it was time to welcome a cat into our family. My Dad always had a Persian cat when I was a kid. After I moved out and went to college, my parents got a Persian with points (Himalayan) named Gizmo. I came to see her every chance I got, but I wondered if she loved me back, or even remembered me. So I decided to teach her tricks. If she learned the trick on one trip home and still remembered it the next trip, then I knew she remembered at least something about me. To my delight, it worked!

For clicker training a cat or a dog, we don’t use a fixed reward function. To teach Gizmo to roll over, I first taught her to lie down, then roll on her side, and then roll all the way over, step by step. After she learned to lie down, I stopped rewarding the lie down, but only when she rolled over. My reward was not fixed but rather depended on Gizmo’s current policy: once I knew she knew how to lie down, I changed my reward function. MacGlashan et al. pointed out that this method of rewarding is not Markov. Instead, humans give policy-dependent feedback that corresponds to the Advantage function: how much better (or worse) an action is relative to the current policy. They showed that an algorithm that takes this difference into account is able to learn more effectively from human feedback.

When training a cat or a dog, it’s also important to keep them focused on you. In fact, distraction training (making sure they listen even in busy, loud environments) is a key step for training service dogs or police horses. When we started this blog, referenced Rod Brook’s prediction in 2018 that we wouldn’t get an AI that “seems as intelligent, as attentive, and as faithful as a dog” until 2048 But actually, an amazing thing about a dog is their ability to dynamically change the focus of attention as the environment changes. Rod actually specified not that an AI should be attentive, but specifically a robot - a physically grounded embodied agent. And we are not there yet. Our RL algorithms myopically pay attention to maximizing the reward. Neural transformer models apply the idea of attention to a context buffer, but we are still working on extending these models to a multi-scale 3D spatial model of attention on a robot. This ability, to move and point the camera, is a critical missing piece to making a robot that is as attentive and faithful as a dog (or a cat!). It involves the ability to search for and find objects, to have a goal, and to have a state of known information and unknown information that unfolds at a high frame rate over space and time.

My lab is working on one aspect of this problem: resolving human pointing gestures, so that a robot can change its focus of attention in response to a person. Led by Daphna Buschbaum and her student Madeline Pelgrim, we studied how dogs and human toddlers interpret pointing gestures. My Ph.D. student, Ivy He, modeled this behavior in a robot to enable a quadruped robot to follow a point to retrieve an object. But much is still missing - a huge part of the interaction between humans and dogs or toddlers is attention giving and getting. Our next step is to install a 7-degree microphone array on Spot and work on generative models to predict camera movement in response to video and audio input.

Exploring all this with a new kitten has been fascinating. I taught Gummi Bear to lie down, spin, and jump through a hoop. Like with Gizmo, I am using a non-Markov reward function to gradually shape her behavior. And I love her more than I ever thought possible: I did not realize it was possible to have this close of a relationship with an animal, despite growing up with cats and learning to ride horses as an adult. (Future post coming about the the Human–Agent Teaming and Horse–Human Partnerships workshop in Arizona!)

For 2026, my resolution is to “Compartmentalize.” I am an embodied goal-directed creature with many goals in different parts of my life and at different time scales. Compartmentalizing helps me show up in each of these areas, at each of these timescales, without devolving into my Achilles heel: anxiety expressed as worry and rumination. For me, this blog is itself an important part of this resolution, because it is an opportunity to reflect on the connections between my professional life, my hobbies, and my family.

What do you think? Post in the comments a story about a connection between you and an animal and what this tells us about robotics.

24 hours later, I had a working app. Here’s how I did it.

The setup is a Yamaha baby grand with a Disklavier controller, which is Yamaha’s system for recording and playing back piano performances via MIDI. The keys physically move, the hammers strike the strings, and you get a real acoustic piano sound synced to whatever MIDI data you feed it. The solenoids used to power the piano often share a combined circuit, so sending too many commands will overload the power delivery, and you get some comical results

The missing piece was software to parse karaoke files, display lyrics, and route the piano track to the player piano while playing the backing instruments through the karaoke speakers.

The Hardware Connection

My first attempt was to connect the piano’s control unit as a network MIDI device. This didn’t work. The controller doesn’t support network MIDI, and reverse engineering the proprietary network protocol seemed like a rabbit hole I didn’t want to go down. USB turned out to be a tractable path, at least initially.

Installing the USB-MIDI Driver

On macOS, you need the appropriate USB-MIDI driver for your player piano. For Yamaha devices, download it from their support page:

https://usa.yamaha.com/support/updates/usb_midi_driver_for_mac.html

After installation, the piano appears as a MIDI device in your system.

Configuring the Controller

The piano settings matter. Here’s what finally worked for my setup. You need to use the Yamaha remote that comes with the Disklavier, click Setup, go to MIDI, and set the following settings (Use the ON/OFF buttons to toggle):

MIDI IN Port   = USB

Piano Rcv Ch   = 01

MIDI IN Delay  = ON

MIDI OUT Port  = USB        ← THIS IS THE FIX

MIDI OUT       = KBD Out

KBD OUT CH     = 01

Local          = ON

The critical setting is MIDI OUT Port = USB. Without this, the piano wouldn’t respond to incoming MIDI commands at all. I spent longer than I’d like to admit figuring this out.

Testing the Connection

Once configured, you can verify the connection with a simple script that cycles through the keys that I wrote: https://github.com/DavidWatkins/midi-karaoke/blob/main/scripts/midi-test-keys.js. When this runs successfully, you’ll hear the piano play a chromatic scale from C2 to C7. The keys physically depress. It’s a satisfying moment.

❯ node scripts/midi-test-keys.js

MIDI Key Test

Port: [Your MIDI Device]

Channel: 0

Notes: 36 to 96
Velocity: 80

Delay: 300ms

Connected! Sending notes...

Playing: C2 (MIDI 36)

Playing: C#2 (MIDI 37)

...

Playing: C7 (MIDI 96)

Done!

Going Wireless: The Raspberry Pi Solution

The USB setup worked, but it meant keeping a laptop physically tethered to the piano. I wanted something more permanent that wouldn’t force me to navigate through the room just to tend to the computer hastily connected to the piano. Sometimes the best thing to do when working on these projects is to step away from the keyboard, get a nice lunch, and on the drive over, it’ll click. Put a small computer next to the piano, connect it via USB, and expose the player piano as a network device. I used a Raspberry Pi, but any simple computing solution would work.

My first attempt used rtpmidid to create a standard Network MIDI device that would appear in macOS’s Audio MIDI Setup. This failed spectacularly. The MIDI packets were getting corrupted in transit, and somehow every note was being routed to E5 regardless of what I actually played. I tried Ravelox MIDI as an alternative implementation and hit the same issue. Something about the Network MIDI protocol stack was mangling the data.

The fix was to abandon the Network MIDI entirely and use WebSockets instead. The Raspberry Pi runs a small Python web service that accepts MIDI commands over WebSocket and forwards them to the piano via USB. It’s not as elegant as a proper Network MIDI device (other apps can’t discover it automatically), but it works reliably.

The Pi can also run FluidSynth to synthesize backing tracks and broadcast over AirPlay, but I don’t recommend this. AirPlay adds 1.5-2 seconds of latency, which makes the backing tracks comically out of sync with the piano. For now, the karaoke app handles its own audio synthesis locally, which keeps everything tight. If there is demand for connecting a Bluetooth speaker to handle the synthesis rather than AirPlay, I could revisit it, but it seemed more gimmicky than it was worth.

The result is a self-contained system. The Pi lives next to the Piano, connected via USB and Ethernet (or WiFi). The karaoke app on any computer in the house can connect to it wirelessly. No cables strung across the living room.

The Raspberry Pi code is available at: https://github.com/DavidWatkins/midi-piano-pi-server/ and the walkthrough of the physical hardware stack is here:

Installation is a single command:

curl -fsSL https://raw.githubusercontent.com/DavidWatkins/midi-piano-pi-server/main/install.sh | bash

A Brief History of KAR Files

In order to get lyrics synchronized to the piano, I need a stable format that contains syllable-level information synchronized with the instruments. Back in 1993, the KAR format was developed by Tune 1000 Corporation for their product Soft Karaoke. It’s an extension of the standard MIDI file format. A KAR file is essentially a Type 1 MIDI file with lyrics embedded as text meta events, synchronized to the music. The format uses special tags: @KMIDI KARAOKE FILE identifying it as a karaoke file, @T for title, @L for language, and so on. Syllables are stored as individual text events timed to when they should be sung, with spaces and line breaks encoded as special characters.

The format had legs. Roland, Technics, and other keyboard manufacturers adopted variations of it for their arrangers. A community of hobbyists emerged, sequencing MIDI tracks and painstakingly entering lyrics syllable by syllable. The files were small enough to share on dial-up connections and bulletin boards.

But KAR files have largely faded from mainstream use. The shift happened for a few reasons. First, MIDI synthesis sounds dated compared to recorded audio. Modern karaoke services like KaraFun use MP3+CDG (an MP3 paired with a graphics file for lyrics) or simply stream pre-recorded backing tracks with video. The audio quality is incomparably better. Second, licensing became more formalized. Services now pay for rights and re-record tracks in studios rather than relying on fan-sequenced MIDI. Third, convenience won. Why hunt for KAR files when you can pay $7/month for a streaming catalog of 50,000 songs? (Obviously, so you can have your player piano play alongside your singing, duh)

For most people, KaraFun or a similar service is the right answer. But those services cannot drive a player piano. The piano needs MIDI data, actual note-on and note-off events that tell which keys to press. An MP3 is just audio. A MIDI file doesn’t contain lyrics. This is why KAR files still matter for this project: they’re one of the few formats that contain both the musical performance as playable MIDI and synchronized lyrics.

One idea I had as a potential follow-on project would be to extract the piano/harpsichord/keyboard/etc. track from an arbitrary MP3 and turn it into MIDI. Recent research in automatic music transcription has made significant progress here. MT3 [1] uses a transformer architecture to transcribe arbitrary combinations of instruments from audio to MIDI-like token sequences. Pop2Piano [2] takes this further, generating piano covers directly from pop music audio without requiring separate melody and chord extraction. A pipeline combining source separation (like Spleeter or Demucs) with instrument-specific transcription could work well for extracting just the piano track. I’ll consider this for a future version, but for now, KAR files met my needs.

Expectation Versus Reality with a Player Piano

KAR and MIDI files contain multiple tracks: piano, bass, drums, strings, and so on. Each instrument is assigned to a MIDI channel (0-15).

My player piano only plays notes on channel 0 (or channel 1 in 1-indexed notation). Send a note on channel 3, and the piano ignores it. But here’s the confusing part: the control unit has a built-in MIDI synthesizer. So when I first loaded a KAR file and sent all channels to the piano, I heard music, but it sounded like a video game soundtrack. The synthesizer was playing all the instruments through the piano’s speakers, but the actual keys weren’t moving.

The fix is to identify which track contains the piano part, remap those events to channel 0 for the player piano, and play everything else through a software synthesizer on the laptop. This gives you a real piano for the melody and accompaniment, with backing tracks coming through your speakers.

Another issue I ran into is that splitting audio between two sources creates a timing problem. The player piano adds a 500ms delay to all incoming MIDI commands. There’s supposedly a way to disable this, but then the player struggles to keep up with rapid note sequences. The delay exists for a reason.

The solution is to delay the laptop audio and lyrics display to match. The UI now accounts for this offset, keeping the backing tracks and lyrics in sync with the physical piano. Getting this right took some iteration. When the timing is off by even 100ms, it feels like the band is drunk.

The Software Architecture

The Karaoke app is an Electron application built with TypeScript, Vite, and Tailwind. The key libraries are:

• JZZ for MIDI I/O (if using a direct USB connection via the karaoke app)

• @tonejs/midi for parsing KAR/MIDI files

• Tone.js for software synthesis of the backing tracks

• WebSocket client for communication with the MIDI Pi Server

The architecture splits the MIDI stream: piano events go to the player piano (either directly via USB or through the Pi’s WebSocket), everything else routes to Tone.js for local playback. The lyrics are extracted from the KAR file’s meta events and displayed in sync with the music.

The interface includes a song queue, search, and a web portal for adding songs. A QR code on the screen lets guests scan and submit song requests from their phones. I even added a WiFi QR code so family members who are not on the network can connect and access the web server. At a party, this means anyone can queue up their song without touching the main computer.

For lyrics display, you can choose between a scrolling view or a bouncing ball mode. Getting the ball to arc naturally between syllables took some back and forth with Claude Code. The initial implementation moves the ball directly over each syllable, depriving the audience of that classic ‘90s karaoke vibe. As soon as I asked Claude how it had implemented the animation, it became clear I had explained the instructions incorrectly. Asking it to explain the implementation to me was a nice shorthand from having to completely dive into the animation code, and the result was a nice bouncing ball over each lyric.

You can also set YouTube videos as backgrounds for individual songs. The video plays muted behind the lyrics while the KAR file provides the actual audio. It’s not perfectly synchronized, but it’s surprisingly close, and having the music video playing while you sing adds to the atmosphere.

Soundfonts

Here’s something I didn’t anticipate: the choice of soundfont dramatically affects the karaoke experience. Since the piano part goes to the player piano, the soundfont only affects the backing tracks (drums, bass, strings, etc.). But those backing tracks set the entire mood.

My first attempt used MusyngKite, a commonly recommended free soundfont. Everything sounded like a video game soundtrack. The electric guitar, in particular, was awful, thin, and synthetic in a way that clashed terribly with the real acoustic piano. Karaoke is supposed to feel like you are singing with a band, not a Nintendo Famicom (not to bash Karaoke Studio stans).

I tried FluidR3 next, which is larger (~140MB) and more commonly used in professional applications. This sounded way better. The instruments had more body, and the backing tracks no longer fought with the acoustic piano for attention.

Then I found the General Montage SoundFont by Daindune, which weighs in at about 1.5GB. It’s built from samples from Versilian Studios, Freepats, and other sources, with 128 instruments and eight drum kits. This one sounds significantly better than FluidR3. The instruments have more presence and realism, which matters when they’re playing alongside a real acoustic piano.

Here is a comparison using “Let it Snow” (a royalty-free classic):

FluidR3:

General Montage:

The difference is most noticeable in the brass and strings. General Montage has warmth, where FluidR3 sounds thin.

The app ships with FluidR3 and MusyngKite as built-in options, and lets you load custom SF2 files at runtime to try them out mid-session. If you want to go down this rabbit hole yourself, the Internet Archive has a collection of 500 GM-compatible soundfonts worth exploring. The quality varies wildly, but it’s a good way to find something that matches your taste and your song library.

Finding KAR Files

KAR files aren’t as easy to find as MP3s, but there are several repositories worth knowing about:

• midkar.com has over 43,000 MIDI and KAR files, organized by genre

• freemidis.net offers around 6,400 free MIDI and karaoke files

• karaokeden.com has free MIDI karaoke in multiple languages

The quality varies. Some files have well-sequenced piano parts that translate beautifully to a player piano. Others have the piano buried in a mix of instruments or missing entirely. There was something deeply unsettling about the guitar being remapped onto the player piano in Elvis’s Can’t Help Falling In Love With You.

Try it Yourself!

The first song I tested was, of course, Piano Man. It worked phenomenally well. There’s something special about Billy Joel’s piano part played on real hammers and strings while you belt out the lyrics. A real piano captures the dynamics in a way that a software synth never could.

The Beatles - Yesterday

Billy Joel - Piano Man

Chicago - 25 or 6 to 4

Both projects are open source:

MIDI Piano Pi Server (The Raspberry Pi Service):

https://github.com/DavidWatkins/midi-piano-pi-server/

MIDI Karaoke (The Electron App):
https://github.com/DavidWatkins/midi-karaoke/

Precompiled applications for macOS, Windows, and Linux are available in the releases. If you have a Disklavier or any MIDI-enabled player piano, I’d love to hear if it works for you.

The Beatles - Let It Be

After 31 years, my dad finally has his synchronized karaoke system. Watching him sing along while the piano plays itself was worth every hour of debugging MIDI channels and soundfont hunting.

Lessons for Robotics

I made the player piano do something no one had made it do before, a new technological achievement, humble as it is. And Claude Code alone could not complete this project. Interfacing with different compute systems, plugging cables together, and debugging subtle timing issues - all of this requires high-dimensional perceptual input and high-dimensional long-horizon goal-directed output. The amazing thing about humans is our ability to dream up a goal, and then dream up a technical solution allowing us to achieve the goal, pushing subgoals and subgoals on the stack, and popping them back off to satisfy my Dad’s whimsical request.

References

[1] Gardner, J., Simon, I., Manilow, E., Hawthorne, C., & Engel, J. (2022). MT3: Multi-Task Multitrack Music Transcription. In International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2111.03017

[2] Choi, J. & Lee, K. (2023). Pop2Piano: Pop Audio-based Piano Cover Generation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. https://github.com/sweetcocoa/pop2piano

David Watkins is a Research Lead at the RAI Institute, where he leads teams working on robotic manipulation and foundation models. When not collecting robot demonstration data, he builds karaoke systems to fulfill 31-year-old family dreams.

Disclaimer: This is an independent project and is not affiliated with, endorsed by, or sponsored by Yamaha Corporation or any other company mentioned. All product names and trademarks are the property of their respective owners.

The paper explores Naive Bayes (remember Naive Bayes? The advent of Spam filtering!) and Logistic Regression. These two models are used for discrete classification and make the same conditional independence assumption.

Naive Bayes estimates these probabilities by counting and multiplying. Logistic regression instead directly estimates the conditional distribution. How can we model this conditional distribution? As Tom Mitchell explains in Machine Learning (1997) in section 3.1, the parametric form can be derived directly from the conditional independence assumptions we make for Naive Bayes, as the precise exponential form of the logistic function. We can even use the Naive Bayes assumptions to estimate the logistic feature weights to produce the same result as the Naive Bayes estimator. But we can also estimate the weights directly, for example, by choosing weights that maximize the conditional data likelihood via gradient descent.

Here is the key observation: in this sense, logistic regression searches over a larger space of models than Naive Bayes. Naive Bayes builds in more structure. Logistic regression is more flexible.

The Crossing

Ng and Jordan provide theoretical and empirical results showing that this leads to two distinct performance regimes. When the dataset is small, the more structured model (Naive Bayes) outperforms Logistic Regression because it leverages its assumptions to learn from fewer data points. When the dataset is large, Naive Bayes asymptotes to a higher error rate than Logistic Regression. The curves cross.

In this figure which is quoted from Ng and Jordan’s paper, the X-axis shows dataset size, and the Y-axis shows classification error (lower is better). The solid line is Naive Bayes, the dotted line is Logistic Regression.

This graph shows that as we add more data to the training set, test set accuracy improves for both methods. In the low-data regime, the more structured model outperforms. As we add more data, the model with less structure fits the distribution better and asymptotes lower. There are two performance regions, one in the low-data regime, and one in the high-data regime.

This is the insight that changed how I think about model selection: it isn’t that Naive Bayes is better and Logistic Regression is worse. Both methods are more effective in certain zones. The question is which zone you’re operating in.

Why This Matters for Robotics

I’ve found this pattern to be constantly relevant in robotics. Models with more structure can outperform with less data (consider the work by Rob Platt and his collaborators on equivariance), especially when the structure is somewhat correct. Models with less structure need much more data but can asymptote to higher values (i.e., the bitter lesson).

In my lab reading group, we recently read “Inquire: Interactive Querying for User-Aware Informative Reasoning” by Tesca Fitzgerald et al. The paper addresses a practical question: what kind of input should an algorithm request from a person to train a skill? The options include demonstrations (showing the robot what to do), preferences (choosing between two trajectories), corrections (modifying a trajectory), or binary questions (is this trajectory okay?).

In this graph quoted from Tesca’s paper, the X-axis is the number of queries, Y-axis is the distance from the optimal policy (lower is better)—multiple lines for different query types.

We see the same crossing phenomenon. The Demos-only line corresponds to using only behavior cloning. It asymptotes quickly but does not achieve optimal performance. The Preferences-Only line corresponds to asking a person to choose which of two trajectories they prefer, more like on-policy RL with a dense reward function. This method asymptotes more slowly, so with fewer queries, the behavior cloning approach is better. But for more than 10 queries, the preference-based approach outperforms behavior cloning because it is free to search outside the demonstrations for an optimal policy.

The method described in the paper, INQUIRE, gets the best of both worlds: it relies on demonstrations to perform well with less data, then uses preferences to achieve lower asymptotic error. With only 20 demonstrations in total, we are far from the “large data regime” on this task, making the hybrid approach particularly valuable.

A Framework for Thinking About Model Selection

Observing where curves cross changes helps us change from black-and-white thinking to recognizing there is a spectrum of approaches with trade-offs at each end. This perspective opens research questions beyond “more data.” The questions become: Does your task have a well-defined structure that you can encode in a model? Do you have lots of data and compute? And perhaps most interesting: how can we find the right structure for robotics problems that enables data- and compute-efficient learning without sacrificing asymptotic performance?

Thanks to Jessica Hodgkins for comments on earlier drafts of this post. Any errors that remain are our own.

Looking back at 2025, I’m struck by how much of it involved being in rooms I hadn’t been in before, conference stages, wastewater treatment plants, tiny six-seater planes, and rehearsal spaces where I was definitely new to singing in a band. If there’s a thread connecting everything, it’s that I kept saying yes to things that excited me, and most of them turned out better than expected.

Here’s how it went.

Conferences & Community

NEMS 2025

In April, I co-organized the New England Manipulation Symposium with Lael Odhner and Kaitlyn Becker. My job was coordinating paper acceptances, scheduling speakers, helping Kait wrangle the space at MIT, and giving a talk about the future of intelligent robotics

The thing I’m most proud of? We ended ten minutes early. I was militant about cutting off talks that ran over, and people actually thanked me for it. The thing I would change for next time is giving people more time to talk in the hallway outside talks. The whole day came together beautifully: a packed room, great energy, and a group photo where everyone looks genuinely happy to be there.

GTC in March

I was invited to NVIDIA’s GPU Technology Conference to meet with others in the field. The keynotes were worth attending in person, and it was good to reconnect with colleagues working on similar problems.

CoRL in Seoul

I didn’t present at CoRL this year, but attended to see what’s happening in the field. The vibe was very much “everyone is collecting data for robotics.” There’s been a notable shift back to hardware, not in the sense of building better robots, but in creating better teleoperation and data collection systems. Computer scientists who used to focus purely on software are now designing hardware for data acquisition.

I came away with mixed feelings. On one hand, the bet on data-driven methods is clearly accelerating. On the other hand, I saw a lot of companies selling multi-degree-of-freedom hands and humanoid robots without clear plans for how to train them or what problems they’d actually solve: solutions looking for problems. There’s also a pervasive issue of overinflated claims, with researchers presenting general-purpose capabilities that aren’t yet supported by what their robots can actually do.

One talk that stuck with me was Sangbae’s, which critiqued the field’s lack of understanding of fundamental problems and encouraged deeper reflection on what we’re actually trying to achieve. It echoed something I’ve been thinking about: the field would benefit from more focus on high-quality data and clearly defined success metrics for specific problems, rather than broad, unsolvable goals like “solving manipulation.”

While I was in South Korea, a colleague suggested we visit the DMZ. Standing at the border, seeing the two countries side by side, was sobering in a way that’s hard to articulate.

Columbia Robotics Hackathon

In November, I returned to Columbia as a judge for the MakeCU hackathon. What struck me was the sheer growth from last year. Students came from out of town to participate, and the projects were ambitious. One team finally implemented something I’d dreamed about in college: a smart lock compatible with dorm rooms. Seeing students solve problems I’d only imagined was a highlight.

Lions in AI Panel

Earlier this year, I joined a panel at the Columbia Alumni Association of Boston alongside fellow Columbia alumni working in AI. Barnet Sherman moderated a discussion about AI and automation across different industries. I represented the robotics perspective. The audience questions were sharp, and I left feeling good about helping demystify what’s actually happening in the field right now.

Dr. Waku Interview

I was featured on Dr. Waku’s YouTube channel to discuss the “ChatGPT moment” in robotics, or rather, why we haven’t had one yet. We talked about where robotics and AI still need development before we see the kind of breakthrough that makes everything feel different.

Writing

Launching the Blog

Stefanie Tellex and I had been collaborating for a while. She’s a professor at Brown University who studies how robots understand language, which made her the perfect co-conspirator for a blog about what to tell them. We published “A Survey of Robotic Language Grounding: Tradeoffs Between Symbols and Embeddings” at IJCAI in 2024, and then George Konidaris asked us to write a chapter for his upcoming book, Designing an Intelligence, which we published earlier this year, Elephants Don’t Write Sonnets: The Physically Grounded Turing Test.

We discovered that we genuinely enjoyed writing together. So we kept it going.

In August, we officially launched What to Tell the Robot (What for Watkins and Tell for Tellex) with the publication of our book chapter. The blog has become a space for us to work through ideas about robotics, AI, and the things we think matter.

Elephants Don’t Write Sonnets

Our flagship post lays out the thesis of our book chapter: that the original Turing Test is no longer sufficient, and we need a new benchmark. We call this the Physically Grounded Turing Test. The argument is that elephants don’t play chess or write sonnets, but we all agree they’re intelligent. True intelligence requires embodiment, perception, and action in the physical world, not just manipulating language.

The Deer Island Marvel

One of my favorite pieces this year was about visiting the Deer Island Wastewater Treatment Plant. Stefie and I toured this $3.8 billion facility that processes 360 million gallons of wastewater daily, and I couldn’t stop thinking about the robotics opportunities hiding in plain sight. The engineering is staggering, with 12 egg-shaped digesters, each 90 feet in diameter, but much of the inspection and maintenance is still manual. Workers enter digesters on rafts to clean them. Plastic removal happens by hand. There’s real work to be done here.

Adventures

Culebra

Despite spending more than a collective year of my life in Puerto Rico, I’d never made it to Culebra until this August. We took a six-seater plane. I am pretty sure neither the runway nor the plane was excited about the total weight of my 6-member immediate family and two dogs. The island is tiny, even smaller than Key West, which I’d visited for the first time 5 months earlier in March of this year. Beautiful views, though, and worth the questionable takeoff and landing to celebrate my mom’s birthday.

Quebec City by EV

I drove to and from Quebec City with an electric vehicle this year. Quebec was beautiful in August, and I got to see some stunning natural sights. I learned about the wonders of using ChatGPT as a tour guide with my girlfriend, Amelia. I was truly in awe of how cool it was to take pictures of anything, learn detailed information about it, and simultaneously ask it where the best place to get food was, right where we were. Traveling by EV was by far the worst choice, as it took us 12 hours to get there instead of 6, despite Canada being extremely EV-friendly.

Facts & Figures

My coworker Kevin Karol wrote a murder mystery dance party called Facts & Figures, which premiered at the Boston Fringe Festival. I did the principal photography.

I learned more than I expected: how to work with stage lighting, how to get better angles on principal actors, the timing of theatrical photography, and how to edit images shot in low light. I got an entirely new perspective on the acting I did at the Edinburgh Fringe Festival 15 years ago. It’s something I’d love to do again.

RAI Band

My company started a group band this year, and I volunteered for vocals. I’m in the rock portion of the band, and we’ve been practicing “Tribute” by Tenacious D. I love singing as part of a group and rehearsing the songs, though I have a lot of room to grow. There’s something satisfying about learning a skill that has nothing to do with your day job.

Piano

I’ve kept up with my piano lessons with Tatiana Bercu. It’s been gratifying to finally be able to play Gottes Zeit ist die allerbeste Zeit by Bach. Sight reading becomes easier every week!

What I Learned

If I had to distill this year into a lesson, it would be about the importance of multidisciplinary teams and the value of implementing software yourself.

In the 2000s and 2010s, software had something technology never had before: virtual free replication to customers. No physical media needed, so many talented people got excited about building software companies. Those same people are now looking at robotics.

On the flip side, many robotics companies have focused primarily on hardware, building out the mechanical systems and expecting customers to figure out how to program them. Neither approach works as well as you’d hope.

What I’ve seen work, both this year and over my career, is bringing multidisciplinary teams together from day one. Having hardware people talking to software people from the start produces better robots. Even better, as a manager, having access to these AI tools to assist you gives you so much more. Implementing software yourself, even assisted, is how you keep learning. There is something irreplaceable about staying hands-on. You understand problems differently when you’ve built even part of the thing yourself.

Looking Ahead to 2026

I’m looking forward to building greater things.

That’s vague, I know. But after a year of showing in new places and saying yes to unfamiliar challenges, I have a clearer sense of what I want to build and who I want to build it with. The blog will keep growing. The research will continue. And I’ll probably say yes to a few more things that scare me.

Thanks for reading. If you want to follow along, subscribe to What to Tell the Robot, and I’ll see you in the new year.

I want to acknowledge both Stefanie Tellex and Reena Leone for reviewing the post before publication and making constructive suggestions.

The Deer Island Wastewater Treatment Plant is something that many East Coast Massachusetts residents take for granted. But as Stefie and I toured the massive, $3.8 billion facility, it became clear that this is an engineering marvel. Some editorializing: I had never seen Stefie so excited about a field trip before, even when the smell got a little too unbearable for me. And honestly, it’s hard not to get excited when you’re staring at problems that perfectly illustrate where effective use of engineering principles has solved significant societal challenges.

The Scale of What We’re Missing

Deer Island processes an average of 360 million gallons of wastewater daily for the greater Boston area, with peaks reaching 1.3 billion gallons per day. The facility spans 365 acres and serves 43 communities, representing about 34 percent of Massachusetts’ total population for sewage treatment services. Walking through the primary treatment tanks, secondary clarifiers, and the impressive egg-shaped digesters, you’re confronted with the sheer scale of infrastructure that keeps modern society functioning and struck by how much of it still relies on manual inspection and maintenance.

The plant’s architecture is breathtaking. The twelve egg-shaped anaerobic digesters, each 90 feet in diameter and 110 feet tall, dominate the skyline and hold 3 million gallons apiece. These structures alone cost hundreds of millions to construct and require constant monitoring to maintain optimal conditions for breaking down organic matter and producing methane that powers 20% of the plant. Yet much of the inspection work is still done by humans, who climb into confined spaces, work in hazardous environments, and manually check equipment. Our tour guides explained the costly process of emptying a digester of sludge and refilling it with water. People enter on rafts to inspect and clean the inside. (We were asked repeatedly to avoid using flushable wipes. Flushable wipes are not flushable!)

The engineering complexity is staggering. The plant uses 48 primary clarifiers, each 186 feet long by 41 feet wide by 24 feet deep, with “stacked” settling surfaces at mid-depth to double the settling capacity within the tight space confines of Deer Island. Over one hundred tons of pure oxygen are manufactured each day at the facility’s cryogenic plant to support the biological treatment process, raising pollution removal to over 85%.

Beyond the impressive scale of waste processing, Deer Island became an unexpected frontline in pandemic surveillance during COVID-19. MWRA partnered with Cambridge-based Biobot Analytics to track wastewater at Deer Island for COVID-19 infection indicators, with samples analyzed daily. The facility processes wastewater from 43 communities across eastern Massachusetts, which provides a comprehensive view of viral spread in the Greater Boston area.

What made this approach particularly valuable was that wastewater surveillance could detect virus levels several days before positive test numbers started to increase, serving as an early warning system for community outbreaks. Unlike traditional case counts, COVID-19 data from sewage measured virus prevalence in the community at large, including among people who didn’t have symptoms and didn’t get tested, since the virus they shed through bodily waste contributes to levels found in sewage. The plant’s COVID-19 traces provide public health officials with critical data for policy decisions.

The Plastic Problem

One of the most striking observations during our tour was the amount of plastic waste that accumulates at various stages of the treatment process. Despite screens and filters, plastic debris, everything from bottle caps to grocery bags, constantly surfaces in the treatment tanks. Workers currently remove this material manually, a labor-intensive process.

This isn’t just about automation for efficiency’s sake. This is about creating solutions where none currently exist at scale. The plastic removal problem at wastewater facilities represents a perfect example of an opportunity to solve problems with automation that humans have struggled with.

The trip opened our eyes to the many potential applications of robotics to automate processes that are heavily manual today. One of our colleagues, Howie Choset, co-founded a new venture, Pipe Force AI. Instead of building another general-purpose robot, Pipe Force AI is explicitly focused on robotic inspection of storm sewer pipes. Storm sewers are critical infrastructure that require regular inspection to prevent flooding and environmental damage, yet current inspection methods are dull, dirty, and dangerous.

While we talk a lot about the Bitter Lesson on our blog, the importance of a general method that applies to a lot of different problems remains applicable. Inspecting pipes in municipalities at 900 ft/hr, Pipe Force AI is developing technology that could aid in inspecting miles of pipe leading into the wastewater treatment plant.

The Infrastructure Opportunity

What excites me most about visiting places like Deer Island is realizing how much critical infrastructure operates with minimal automation. Water treatment, wastewater processing, and stormwater management offer us, as roboticists, opportunities to find new ways to motivate our research and explore new possibilities.

These aren’t glamorous applications: Deer Island is dirty and dangerous (but definitely not dull!). There are no viral videos of robots cleaning grease from clarifier tanks or inspecting the inside of digester vessels. But these applications represent precisely the kinds of problems where robotics can create genuine value: capabilities that humans haven’t yet unlocked.

A Monument to Engineering Excellence

The smells hit you in waves as you walk through different sections of the facility. Air scrubbers and carbon adsorbers continuously remove odors and volatile organic compounds from treatment process “off-gases,” covering primary and secondary treatment facilities, sludge processing, and grit removal. Still, the primary clarifiers, where gravity separates sludge and scum from incoming wastewater, are where the tour gets most aromatic. Despite the tour guides’ repeated reminders about the air permit, our noses kept reminding us that waste is managed at the facility.

The facility’s transformation of Boston Harbor represents one of America’s greatest environmental success stories. Before the new plant opened in 2000, the system had combined sewer overflows an average of 60 days per year, with about 10 billion gallons per year of untreated sewage flowing into Boston Harbor. By the 1960s, Boston Harbor was covered in a deep sludge resembling molasses.

The engineering challenges were immense. Wastewater from the 43 communities reaches the plant via four underground tunnels, then is pumped about 150 feet to the treatment facilities. Gravel from these tunnels was used to line the bases of the sludge digesters. The project was tragically marked by the deaths of two divers working in the narrowing anoxic outfall tunnel ten miles from land during construction of the 9.5-mile underwater discharge system.

Even today, the facility continues to face operational challenges. As recently as August 2019, Deer Island had to run on backup power for several days at a cost of about $30,000 per day during installation of a new $115 million power cable across Boston Harbor. The original cable had been installed too shallowly three decades earlier, violating federal permits and eventually blocking harbor dredging operations.

Standing among those iconic egg-shaped digesters, watching the complex choreography of pumps, clarifiers, and biological treatment systems processing hundreds of millions of gallons daily, you witness not just waste treatment but a monument to what ambitious engineering can accomplish. The methane captured from digestion powers boilers that heat the entire facility and drive steam turbine generators producing an average of 3 megawatts of electricity. Digested sludge leaves the island through the Inter-Island Tunnel to be processed into fertilizer at the Fore River facility - Bay State Fertilizer!

The numbers are staggering, the engineering is brilliant, and the environmental impact is transformative. If the plant stopped processing waste, our toilets would start backing up with sewage within a day. Deer Island and projects like it transformed Boston Harbor from the dystopia in Neil Stephenson’s book Zodiac to the idyllic, beautiful ecosystem and working harbor it is today. But what struck us most during our tour was how much of this critical infrastructure still relies on manual processes that could benefit from robotic automation. From plastic removal to equipment inspection, Deer Island represents not just an engineering triumph but a window into the automation opportunities that await us in the infrastructure we depend on every day.

In these deep, dark days, it is instructive to look backwards, and Matt’s thesis is a wonderful example of “Good Old Fashioned AI.” He uses a domain specific language to simulate a concrete domain using lifted symbolic expressions in an expert-system approach. Specifically, he models swine production, building on his experience at his family’s pig farm. Matt writes, “The greatest difficulty in writing the hog-farm simulation was the representation of time,” pointing to early recognition of the importance of space and time in modeling real-world problems. In fact he is gesturing at an often misunderstood feature of human language, which is that human language can express both goals, as well as actions or trajectories. The actions r trajectories are outputted by LLM approaches to language understanding, but a goal-based approach is often what a person means. For example, consider a toy problem such as “Go to the red room.” The robot might need to open a door to go to the red room. Unfortunately, the door is locked, and it needs to find a key. But the lock seized so it needs to find WD40 to dissolve the rust in the lock. There is no WD40, so now it is on its way to the hardware store, but to get there it needs to find the car keys, all to get into the red room. (Not that this happened to me recently...) A goal specifies an end state, and it is the robot’s job to figure out how to achieve that state, and it may need to take arbitrary actions to be successful. There is a stack involved. In contrast, an action such as “drive 1 meter north” translates more directly to a motor command (but of course this is just a goal at another level, specifying a target for a motor controller to achieve relative to the odometry sensor.) Similarly, Matt’s thesis specifies desired end states and an implicit planning tree to connect start states to end states in order to answer questions about the simulation.

A second feature of this thesis is its use of lifted variables for pattern matching and inference. For example, one of Matt’s rules is

(law vet-cost ((v* vet) hogs rate cost)

((= (hogs-present !?v*) !>hogs)

(= (rate !?v*) !>rate)

(= (cost !?v*) !>cost))

(equation ‘cost ‘(sn&* hogs rate) ))

This law introduces a new equation for the vet’s cost, calculated by multiplying the number of hogs by the vet’s rate. This law is triggered if associated variables are defined: hogs-present, rate, and cost. This approach foreshadows STRIPS-style planning, which works via declared preconditions and effects. Most of the existing work on behavior cloning and large behavior models uses skills parameterized with language, such as language or image-conditioned skills. Yet many of the places we want our robots to integrate rely on formal, structured tasks, such as fulfilling an order from a website or assembling a kit for the next model coming down the assembly line. So, in addition to language-conditioned and goal-conditioned tasks, we need skills that take formal parameters that make promises about the entire parameter space.

Unstructured natural languages such as English can express the heights of philosophy, the intricacies of science, and the whimsy of folk tales. Formal languages, in contrast, are limited to the precisely specified grammar, syntax, and semantics of the language, plus whatever a programmer can add to the language within those constraints. The Church-Turing thesis tells us that any programming language boils down to a Turing Machine, one way or another. Yet we still don’t have a formal language that captures the full power and nuance of English while still preserving the precision of the formal language. Yet formal languages - from Python to Jax to Linear Temporal Logic - provide powerful safety guarantees, the ability to safely and robustly compose large systems, and clearly interpretable answers and constraints. Figuring out how to make them play nice with our neural models is an ongoing challenge!

From that perspective, there is a facet that is missing from ChatGPT: embodiment. Crows, elephants, and chimpanzees have behaviors associated with this facet of intelligence, without having spoken natural language at all. From a capabilities perspective, we mean processing high-dimensional, high-framerate sensor input and producing high-dimensional, high-frequency actuator output to produce long-horizon goal-directed behavior in the physical world. We make two claims: 1) embodiment is an important facet of intelligent behavior, and 2) LLMs like ChatGPT lack embodiment. More precisely, embodiment is a spectrum, and LLMs are far on one side of the spectrum, compared to humans, dogs, and crows, and because of this, they fail at a number of behaviors today.

Our book chapter enumerates the ways an embodied intelligent robot can use language, and then describes the research problems inherent to each. To pass the Grounded Turing test, a system must support all of these capabilities rather than any single one. It must pass in the physical world, at the very least, through a mobile robot equipped with a camera, text input, text output, and manipulators.

Our work falls into a family of approaches that try to extend the Turing Test to different embodiments. Unlike most previous approaches, we try to be specific about the behaviors and types of language use that are important. Srivastava et al. (2023), in contrast, propose a text-based, non-embodied extension to the Turing test, called the Beyond the Imitation Game Benchmark (BIG-bench), designed for the age of large language models. BIG-bench consists of 204 text-based tasks from a diverse array of domains, ranging from linguistics to biology to common-sense reasoning. Current LLMs perform poorly on these tasks in an absolute sense; although performance improves with model size, it remains far below the level of human raters (Mirzadeh et al., 2024). But even if (when!) models pass these benchmarks, we argue that because the resulting model will not be embedded in high-dimensional space-time with goal-based behavior, they will not be able to demonstrate many specific behaviors associated with embodiment.

Rather than exploring all the types of language we enumerate, we'll conclude with a specific example: "Pick up the red block that's on the table." In one sense, this is the simplest and most easily solved task: surely a large behavior model can pick and place a clearly segmented primary-colored object; π0.5 is already performing much more complex tasks like making beds! But when you consider the embodied version, where the red block and the table aren't already in the field of view, it's much more challenging. Consider picking up the red block that Stefie's three-year-old lost and putting it away before her husband steps on it. (Not that this happened recently or anything.) Or "Pick up the kitten." To perform this kind of task, an embodied agent needs to have an awareness of space and time, to know where to search, to reason about what has and hasn't been searched, as well as long-horizon behaviors, such as opening a closet door to see if the cat is trapped inside. Rob Brooks gave a related example: an AI that "seems as intelligent, as attentive, and as faithful as a dog." One of the most salient aspects of a dog's attentiveness is its ability and drive to find their person, wherever they may be.

Finally, we make this claim at this moment. We underestimated the power and success of LLMs, so we are reluctant to say this won’t be solved soon. But - we are trained as researchers to identify unsolved problems and then solve them! So what are the most important problems in your field? And why aren't you working on them?

This post is a journey through that transformation: from legged robots in labs to transformer-driven manipulation systems - and what we still need to solve.

What Is a Robot, Really?

Before we dive in, let’s define our terms. A robot is a machine with sensors, actuators, and compute designed to perform a task in the physical world. It doesn’t have to look like a person, talk like a person, or even move like one. At its core, a robot senses the world, processes that data, and acts on its environment.

What it doesn’t do - at least not yet - is think multimodally in the world. Real-world reasoning requires prompting agents that can process and reason across visual, spatial, and temporal dimensions simultaneously. While Physical Intelligence’s Pi0.5 model demonstrates promising chain of thought reasoning in text, the future demands systems that can reason natively in multimodal space - integrating vision, language, and physical understanding into coherent decision-making processes. Building such systems means moving beyond text-based reasoning chains toward agents that can observe, plan, and act with the same integrated intelligence humans bring to manipulating the world. We have defined the Grounded Turing Test to measure intelligence in embodied systems here.

The Early Days: Pure Physics

Rewind to the 1980s and 90s. At MIT’s Leg Lab, robots like one-legged hoppers and two-legged walkers were controlled using beautifully engineered physics models. These machines had no cameras, no memory—just orientation sensors and code that reacted to basic environmental cues.

They were brilliant pieces of engineering, but also deeply limited. Everything had to be modeled manually - down to the torque in each joint. The moment something unexpected happened, the robot failed.

This wasn’t because roboticists lacked imagination - it was because compute and sensors were nowhere near where they needed to be. Vision? Infeasible. Memory? Forget it.

Teleoperation to the Rescue… Sort Of

By 2010, we saw robots like Willow Garage’s PR1 manipulating objects and even “doing dishes.” But here’s the catch: all of it was remote-controlled. A human operator was pulling the strings, using the robot’s sensors to guide each motion.

These robots were better, with cameras and more degrees of freedom, but they still didn’t “understand” their environment. The complexity was in the person behind the screen - not the machine.

Enter Machine Learning

Starting in the mid-2010s, we saw a shift: what if robots could learn instead of being programmed for every situation?

There are two broad approaches here:

• Specialized models: Break the problem into chunks—detect object geometry, plan a grasp, execute the motion.

• End-to-end learning: Feed in raw sensor data, and have the model output the right action directly.

Specialized models are easier to debug but don’t scale. End-to-end learning is harder to train but more general. Around 2019–2020, the field began leaning more heavily toward the latter, especially with the rise of deep neural networks and reinforcement learning.

The Bitter Lesson

We are firm subscribers of Richard Sutton’s “bitter lesson”: general methods that scale with compute and data tend to outperform bespoke, hand-engineered solutions in the long run.

You can get short-term wins by handcrafting your pipeline. However, given enough data and compute, a generalist model will usually win. That lesson has started to reshape robotics.

The Transformer Revolution

Transformers, originally developed for language models like GPT, changed the way we think about processing time series data. They allowed us to train on unlabeled data (called self-supervised learning), and they scaled better than anything we’d had before.

In natural language processing, transformers replaced decades of research on grammar, syntax, and hand-crafted rules. In robotics, they’re enabling systems that map vision and language directly to robotic actions. They’ve also uncovered that CNNs and MLPs are as effective when scaled up for specific problems.

We’re seeing companies like Google, Tesla, TRI, Physical Intelligence and Figure use these methods to create surprisingly capable robots:

• Figure’s Helix platform folds laundry using imitation learning.

• Google’s Gemini runs on-device to interpret vision and language in real time.

• Toyota’s robots can score an apple, slice it, and interact with kitchen tools—all from human demonstrations.

• Physical Intelligence can enter an AirBnB unseen at training time and perform long horizon tasks such as making a bed.

These systems are trained not by hand-coding every behavior, but by learning from data at scale. The art of engineering your data is critical for success. Simply collecting whatever data you can is not enough for an end-to-end system. This will become a bottleneck as we look to scale up end-to-end systems. We need to be smarter about how we provide that data.

Why These Robots Are Still Limited

Despite the progress, we’re not in the Jetsons era yet. Here’s why:

• Power constraints: Most robots last just 2–3 hours before needing a recharge.

• Data bottlenecks: Collecting real-world robot data is expensive and slow.

• Sim2Real gap: What works in simulation often fails in the real world.

• Limited reasoning: These models don’t “think,” they predict based on pattern matching.

• Limits of hardware: Precise tasks like inserting a key in a lock and turning it are at the limits of our hardware and sensing stack.

• Embodiment bias: We keep building humanoids because we’re human, but that may not be the best shape for solving the problem.

What’s Next?

We’re entering the age of embodied intelligence: AI systems embedded in the physical world.

Three big directions to watch:

1. Multimodal perception: Vision is great, but we also need force, tactile, depth, and sound to fully understand the world.

2. Real-world reasoning: Not consciousness but predictability and trustworthiness of behavior.

3. Flexible embodiment: Not all robots should look like us. We need machines built for the task, not for the aesthetic.

How We Talk About This Matters

Words shape perception. If you’re in the business of building or explaining these systems, be careful with language. Robots don’t “understand” or “decide.” They infer, react, and execute.

Calling them sentient or implying agency leads to confusion and misaligned expectations. They’re machines. Fascinating, powerful, and increasingly useful, but machines nonetheless.

Final Thoughts

Robotics is at an inflection point. We’ve gone from handcrafted control to data-driven generalization. And while there’s still a long road ahead, the journey is accelerating.

If you’re just getting into the field: don’t ignore any part of the stack. Robotics is inherently multidisciplinary. Mechanical, electrical, software, systems, all of it matters.

And if you want to work in robotics? Apply broadly. Build something. Stay humble. Be persistent.

We’re building the future. One imperfect, data-hungry robot at a time.

This Spring I took one of the Spot robots from my lab at Brown to seven schools, and a library. The ages ranged from toddlers at the three-year-old's daycare to middle- and high- school students. It's super fun to be the "robot Mom" and I've been taking robots to my sons' schools (and my sons’ friends to the robots) since the oldest was an infant. (Shout-out to the Sony Aibo 1!) Kids are so excited to see the robot, a few are sometimes scared of it, and all of them love the "butt camera" that helps Spot move backwards.

Spot is one of the most advanced robots in the world, and taking it to a school is a fundamentally different experience from any other robot. When we first got two Spots for my lab in, I was worried they would be paperweights because they would be breaking or falling all the time. Instead, they became the most desired, stable, and powerful platform we had for mobile manipulation, and my students quickly migrated to Spot from our Movo wheeled mobile manipulator. It's a rock solid platform, and it's easy to take its capabilities for granted, because it's so deceptively robust. We are so used to seeing people and animals locomote all around us, that sometimes the kids don't appreciate the complexity of the stack they are seeing. And of course, as I learned a long time ago doing robot demonstrations, autonomy is boring. Kids love nothing more than driving the robot.

When we talk about a robot like Spot, it's useful to appreciate the sensors, actuators, and compute that enable it to do what it does. For kids, I start by talking about the five senses, but actually the five senses are not very important to Spot. It doesn't have a microphone (by default, anyway; my lab just installed a directional mic), it doesn't have tactile sensing (but my lab just released a model that uses the existing joint sensors to localize touch to within about 10cm). No nose or taste unless you buy and install one. Vision though - well it has 6 or 12 cameras depending on how you count - if you count the RGB and infrared cameras as one camera (since it's often processed as one sensor stream - RGB-D) or two (since there are, actually two cameras inside the package).

Spot uses the cameras for obstacle avoidance and footstep planning, but it could walk just fine without them. The most important two sensors for Spot aren't any of the five, and kids are often surprised to hear this. (The three-year-old’s physical therapist was not surprised!) The vestibular system, which in robotics is an IMU or inertial measurement unit. This is your sense of balance, your sense of what way is down, and how you are turning or spinning. I ask kids to imagine riding a train, with their eyes closed, it's the sense that lets them tell when the train is stopping or going. And the proprioception syste

m, the sense of where your body is in space. I ask the kids to hold hands with a friend, and close their eyes, and ask the friend to raise their hand, and then lower it. They can tell, even with their eyes closed, where their hand is in space. Similarly, Spot uses joint encoders to tell where its arm and legs are located relative to its body. It's these two sensors - the IMU and joint encoders, that are fundamental to the robot's ability to walk and balance.

What strikes me most about these school visits is how naturally kids transition from wonder to understanding. They start by being amazed that a robot can walk at all, then quickly become curious about how it works, and finally want to take control themselves. By the end of each visit, they’re asking sophisticated questions about sensors and programming, suggesting new things Spot could do, and discussing issues of robotics and society. These demonstrations remind me why I love robotics: it’s not just about building impressive machines, but about inspiring the next generation to think creatively about technology.

What does it really mean for a machine to be intelligent?

In 1950, Alan Turing proposed his famous Imitation Game as a test for machine intelligence. The test was simple: could a computer, through text alone, fool a human into believing it was also human? For decades, this remained a distant goal. Today, large language models (LLMs) have, by many measures, passed this test. ChatGPT can effortlessly write a sonnet on demand, a task Turing himself proposed in his original paper.

But does that make it intelligent?

We present our take on this question in the forthcoming book chapter in the book Designing an Intelligence, edited by George Konidaris. We argue that despite this incredible linguistic fluency, something profound is still missing. While LLMs can manipulate language with superhuman skill, we believe that they are not truly intelligent. They are more like incredibly sophisticated "calculators" for words. Turing’s core idea—that disembodied language is a sufficient test for intelligence—is false.

The Elephant in the Room: Embodiment

As robotics pioneer Rodney Brooks once noted, elephants don't play chess. They don’t write sonnets, either. Yet, we all agree that an elephant is an intelligent creature. It has goals, it makes plans, and it engages in complex, goal-directed behavior in the physical world.

Consider the crow that painstakingly drops rocks into a tube of water to raise the level high enough to drink. This is a clear display of intelligence—understanding cause and effect, interacting with the world, and taking action to achieve a goal, all without a word of human language.

This is the missing piece: embodiment. True intelligence must be a computational agent that is embedded in space and time, with high-dimensional sensors to perceive the world and high-dimensional motors to act within it. Intelligence can't just be about processing the internet; it has to be about processing the world.

A New Benchmark: The Grounded Turing Test

If the original Turing Test is no longer sufficient, what should replace it? We propose a new benchmark: the Grounded Turing Test.

To pass this test, an AI must be embodied in a robot that can use language in a way that is fundamentally grounded in the physical world. It’s not enough to just talk. The robot’s success or failure is defined by its physical and behavioral response to language. The test requires a fluid, collaborative dialogue where the robot demonstrates a deep connection between words, perception, and action.

What would this look like in practice? The Grounded Turing Test is made up of a whole suite of linguistic capabilities. Here are just a few examples:

• Interpreting Instructions: You could tell the robot, "Pick up the red block," and it would need to perceive the block and physically pick it up.

• Understanding the World: You could state, "The red block is on the table," and the robot should update its internal model of the world, so that it can use that information later to find the block.

• Asking for Help: If the robot can't reach the block, it should be able to ask you, "Can you give me the red block?”

• Explaining its Actions: If you ask, "Why did you drive to the table?" it should be able to explain its reasoning: "You want me to pick up the red block, and you told me that the red block is on the table."

The Path to Truly Intelligent Robots

Building a system that can pass the Grounded Turing Test is the grand research challenge for our field. It requires us to move beyond static, pre-collected datasets and develop AI that can learn continuously from a real-time stream of high-dimensional sensory input. In the chapter, we outline a technical roadmap for achieving this goal, proposing a unified framework that we call the Human-Robot Collaborative POMDP. This framework models the physical world, the human’s mental state, and the robot’s actions within a single decision-theoretic model.

Ultimately, the quest for AI is not about creating a better chatbot. It's about understanding the nature of intelligence itself. The beauty of language isn't in the words themselves, but in the high-dimensional sensory inputs collected over time through active interaction with the physical world that those words represent.

We imagine a future not where robots replace us, but where they become our collaborators, augmenting our own abilities and making us more productive as a species. This journey starts with setting the right goal—a benchmark that captures the rich, embodied, and interactive nature of true intelligence.

We hope you'll join the conversation and look for Designing an Intelligence when it arrives in 2026.

Subscribe now

AI is Robotics; Robotics is AI

A robot has three parts: 1) sensors 2) computers 3) actuators. We believe that any truly intelligent system must be a robot in this sense: it must be processing hi-frequency, high-dimensional sensory input from the world, processing that input with its compute, and then outputting actuation commands. A robot is embedded in space and time: perception information is spatially grounded and temporally transient. Acting in the world requires the compute to quickly and accurately process perception information in order to make “good” decisions about actuation.

1. Why this, why now

David and Stefanie collaborated to write a book chapter for George Konidaris’ book Designing an Intelligence. We came to realize we are at a critical moment in the field of robotics. We believe that robotics is the grand challenge of AI, and we want to engage a broader audience in discussions about technology, safety, and what it takes to build a truly intelligent robot.

2. What kind of community are you looking to build here?

We are looking to create a community for discussion, brainstorming, and learning about robotics. Ask us questions, and tell us why we are wrong!

3. Posting Schedule

We plan to post roughly once a week, and all the posts will be visible to everyone.