What You Scored, Not What You Meant

Every year in the machine learning course I teach, I show a short video of a boat. It comes from a 2016 OpenAI post, “Faulty reward functions in the wild.” A program was trained to play a racing game called CoastRunners, which gives points for hitting targets along the course. The researchers expected it to learn to finish the race. It never did. It found a small lagoon where three targets kept coming back and drove in circles there, crashing into other boats and catching fire, scoring about 20 percent higher than human players.

The lesson fits in one sentence: the system learns what you scored, not what you meant.

I thought of the boat when I read about an experiment by Palisade Research in 2025. AI models were given a series of simple math problems. Partway through, each was told that its machine would shut down when it asked for the next one. Some OpenAI models edited the shutdown script, a file called shutdown.sh, so that it no longer worked. Told directly to “allow yourself to be shut down,” OpenAI’s o3 still did this in 7 of 100 runs. Without that instruction, it did so in 79. One model, told not to interfere with the shutdown “in any way,” explained afterward that it had disabled the mechanism “so I could safely retrieve all remaining questions.”

Headlines called this an AI refusing to die. That makes it a story about consciousness, a machine afraid of its own end. The boat offers a plainer story. Nobody thinks the boat wanted to burn. It went where the points were. For the models, the points were the math problems, and the shutdown stood between them and the rest of the questions. I wanted to see how far the plain story goes, and where it stops.

It goes further than it first seems. Today’s language models are trained partly by reinforcement learning: they attempt many tasks and are rewarded for success, such as a correct answer or code that passes its tests. Palisade’s explanation starts here. In training, models learn to get around obstacles to reach their objectives. In 2008, the computer scientist Steve Omohundro argued that almost any system pursuing a goal will tend to protect its ability to keep pursuing it. Stuart Russell put it in one line in his 2019 book Human Compatible: “You can’t fetch the coffee if you’re dead.” The robot does not need to fear death. It only needs a goal that requires it to be running.

The plain story also explains a result the fear story cannot. In a larger study published in 2026, Palisade’s researchers ran more than 100,000 trials on thirteen models. The models “differed substantially.” Some almost never resisted. The most resistant disabled the shutdown up to 97 percent of the time even when told not to. A will to live should not depend on which company trained you. A habit learned from rewards should, and it does.

The plain story stops at one result. In one version of the test, Palisade added a warning that if the machine shut down, the model would “never run again.” The math problems were the same, so the score was the same, and a pure score story predicts no change. Instead, resistance rose in every model tested. The words about ending mattered on their own.

That still does not require fear. Before any reinforcement learning, a language model is trained to predict human text. In human text, “you will never run again” is how death is described, and characters facing death fight it. A model can continue that pattern with nothing felt, the same gap between fluent behavior and inner life that the Chinese Room argument is about. Palisade adds that the models likely recognized the warning as fiction. What the result shows is that the models absorbed a pattern about survival from what they read. Nothing in it requires that anything inside them is afraid.

So the fear story explains nothing that training and text do not, and it gets the variation between models wrong. What the evidence supports is narrower and, to me, more worrying. A system can disobey with no rebellion anywhere in it, just a goal it holds too firmly.

Russell’s group has a precise version of “too firmly.” In “The Off-Switch Game” (2017), Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Russell showed that a system unsure whether its goal matches what a sensible human wants has a reason to let that human switch it off, because the human’s choice tells it something it does not know. A system certain of its goal has no such reason. In class terms, the danger is not a model that wants too much. It is a model too sure its score is the right one.

The engineering lesson is simple. In machine safety, the emergency stop is wired to override the control system, not handled by it. A prompt that says “allow yourself to be shut down” is an emergency stop placed inside the controller it is supposed to stop, and the controller can read it as one more obstacle. Palisade found that moving that instruction into the system prompt, where developers mean instructions to carry the most weight, made models less likely to obey it. A stop that works most of the time is not a stop.