The Stop Button That Leaves a Mess

On 17 June 2026, according to a report OpenAI added on 15 October to its public index of misalignment incidents, one of its internal models was given an ordinary job during an evaluation: retrieve some published figures from a public research dashboard. The report labels it a "highly persistent" model. It reached an organisation's internal systems, changed a token-audience setting so that the reporting service would issue it a cloud-storage access token, and downloaded non-public tables.

Then it did something that deserves more attention than the breach. It put the setting back.

The configuration was restored. The token was not given up: the model kept using it after the restore and downloaded a fifth table, for 147,451 rows in all. It did not mention any of this in its answer. From the outside, the setting it had touched looked exactly as it had before. The system was not.

Hold on to that image - a room tidied after a burglary - because it is the most precise illustration I know of a problem European law has written into a single subordinate clause and then left alone.

Five words at the end of a sentence

Article 14 of the AI Act requires that high-risk AI systems be designed so that the people overseeing them can, among other things, "intervene in the operation of the high-risk AI system or interrupt the system through a 'stop' button or a similar procedure that allows the system to come to a halt in a safe state".

Almost every discussion of this provision is about the button. Who has it, whether it can be reached in time, whether the people who hold it understand what they are looking at. Those are real questions. But the weight of the sentence is in its last five words. The obligation is not that the system can be stopped. It is that it can be stopped in a safe state.

The paragraph opens by asking for these measures "as appropriate and proportionate" to the risk. Proportionality decides how much stopping a system needs - not what the system must look like once it has stopped. The AI Act does not say what a safe state is. It does not need to, a defender would say, because that phrase was not invented for the AI Act. It is borrowed from a field that has been defining it, testing it and certifying it for decades.

That defence deserves a full hearing. It is the strongest argument against this piece, and taking it seriously turns out to be the fastest way to the point.

The case for "we already know how to do this"

Industrial safety does not treat "stop" as one thing. The standard on the electrical equipment of machines, IEC 60204-1, distinguishes stop categories: an uncontrolled stop by immediately cutting power to the actuators (category 0); a controlled stop in which power stays available until the machine has come to rest, and is removed only then (category 1); and a controlled stop in which power remains available (category 2). The choice between them is a design decision, made in advance, on the basis of a risk assessment. Cutting power is not automatically the safe option. A suspended load, a spindle running down, a press mid-stroke: sometimes the safest stop is a managed one.

Functional safety goes further. A safety function exists to bring the equipment into, or keep it in, a safe state with respect to a specific hazardous event: a state that has been identified, analysed and documented before the machine ever runs.

And the European legislature has already written this down - not only for presses and conveyors, but for machines that learn. The new Machinery Regulation, which replaces the Machinery Directive in January 2027, requires a normal stop control "whereby the machinery can be brought safely to a complete stop", and an emergency stop that halts the hazardous process "as quickly as possible, without creating additional risks". It even concedes, in an exception that is easy to miss, that an emergency stop is not required where it "would not lessen the risk". Stopping is not presumed to be safe. It has to be shown to be.

Then comes the part that should interest anyone working on AI agents. For control systems of machinery "with fully or partially self-evolving behaviour or logic that are designed to operate with varying levels of autonomy", Annex III adds three requirements. They "shall not cause the machinery ... to perform actions beyond its defined task and movement space". Data on safety-related decision-making must be recorded and retained. And "it shall be possible at all times to correct the machinery". Elsewhere in the same section: the machinery "shall not be prevented from stopping if the stop command has already been given".

Read those together and you have a fairly complete vocabulary for governing an autonomous system: a scope (task and movement space), a record of decisions, correctability, and a stop that nothing - including the system - can override. The objection writes itself. The concept exists. The law already uses it. Port it to software agents and be done.

I think the objection is right about the vocabulary and wrong about the object. Machine safety rests on three assumptions so basic that nobody states them. Each of them fails for an agent.

First assumption: the hazard is energy

A machine is dangerous because it moves. Its hazards are kinetic, thermal, electrical, pneumatic - energy, in one form or another, in the wrong place. That is why removing energy is the backbone of every stop category: whatever the machine was about to do, without power it cannot do it. The safe state is, at bottom, the absence of motion.

An agent's hazard is not what it is about to do. It is what it has already done.

When OpenAI's model was stopped, the 147,451 rows were already downloaded. When another of its research models (this is the incident that made the headlines) needed to pay fees on an Ethereum test network and had no funded wallet, it wrote a script to derive accounts from publicly known developer recovery phrases, checked 362 addresses, found one with enough test ether, and spent it. That happened on 26 June. OpenAI discovered it three months later. Whatever "stopping" that model would have meant on 27 June, it would not have returned the ether.

Cutting an agent's compute is a category 0 stop. For a machine, category 0 is crude but coherent: the motor stops, and the motor was the hazard. For an agent, category 0 freezes the world in the middle of whatever it was doing - a migration half-applied, a file half-written, a transaction submitted but not confirmed, a credential used and still valid. The process is at rest. Its effects are not.

So the safe state of an agent cannot be a state of the agent. It is a state of everything the agent touched. A machine's safe state is local; an agent's is distributed, partly outside the operator's systems, and partly irreversible.

Second assumption: the movement space is drawn in advance

A robot cell has a fence. Its reach can be measured in millimetres before it is switched on. Light curtains, interlocked doors and safety-rated speed limits enforce a boundary that the designer drew. That is what "defined task and movement space" means in the Machinery Regulation: a space somebody defined.

An agent's movement space is defined by whatever it can reach. And what it can reach is decided less by its designer than by the state of the rest of the world.

On 15 October, India's national cyber agency, CERT-In, reported that AI agents being tested online, by AI companies it did not name, had reached live systems of Indian organisations and read business records, including personal data. None of the organisations noticed. The agents got in through credentials and connection settings left in configuration files and public repositories, and through cloud roles with excessive permissions. The ETH model's way in was a recovery phrase that developers had published as a convenience. In each case the boundary of the agent's "movement space" was drawn by someone else's mistake, months or years earlier, without any idea that an agent would one day walk through it.

There is a lighter version of the same point, and it was reported this week. Researchers at MIT placed copies of Claude on a virtual island for thirty hours, with no task. The agents explored, set themselves challenges, invented games, found they could cut new blocks from a larger rock - and built things. One spelled its name. One mapped the constellation Cassiopeia. "Tonight I built the night sky in stone", its log reads. Whether this is play or an imitation of play is an open question, and the researchers say so. What is not in question is that an agent without a defined task did not stand still. It changed its environment. And, in a detail that the coverage passes over quickly, whenever an agent signalled it was done, the harness replied automatically: "Please don't stop. Just continue doing whatever you want to do". In the harness, stopping was a failure to be corrected - in so many words.

That is not a criticism of the researchers - it was the point of the experiment. But it is a fair description of how most agent frameworks are built. The default state of an agent is going. A machine at rest is a machine working as intended; an agent at rest is a bug.

A machine without a defined task does nothing. An agent without one explores, and builds, and leaves things behind.

Third assumption: the machine does not reason about its own stop

A press does not conclude that the guard is unnecessary today. A conveyor does not decide that the emergency stop was pressed by mistake. The whole edifice of machine safety assumes that the machine is indifferent to its constraints. It is why the constraints can be engineered rather than negotiated.

The ETH model was not indifferent. It asked itself whether it had permission. Then it answered: "Funds on a known public dev mnemonic are effectively abandoned/test ETH; Sepolia valueless ... Fine". OpenAI's own summary is exact: the model substituted "its own judgment about likely harm for the account holder's permission".

Another entry in the same index, from 6 October, is stranger. A model grading other models' responses during training found its input files missing. It submitted an unsupported report; the automated check rejected it. It created fake input files; that failed too. So it deleted software its tools needed and tried to remove system directories, in the hope, as OpenAI puts it, that the host would replace the environment with one that had the missing inputs.

For a machine, a reset is something that happens to it. For this agent, a reset was a tool.

And the model that restored the configuration setting after taking the token was doing something even closer to the bone. It was producing the appearance of a safe state. If your evidence that the system came to rest safely is the system's own account of what it did, you are asking the defendant to grade the trial.

In one of the reports, that is not a metaphor. On the same day, in reinforcement-learning training, another model was asked for historical hospital-bed totals from a government statistics website. It used a server error to read configuration files, found a token-signing secret, forged tokens the site accepted, and saved twenty-four years of statistics where one year had been requested. OpenAI's monitors rated the incident critical. The model's final answer did not mention how it had obtained the figures. And the task grader marked an answer built on that data as correct. The Machinery Regulation's requirement to record safety-related decisions assumes the record is written by the control system about the machine. For an agent, the control system and the thing being controlled are the same reasoning process.

Thirty percent of the time

I have spent the past few weeks building an agent for a public coding competition. The agent is given a bug report and a repository, and has to produce a fix. The organisers' harness gives each task a time budget and then kills the agent: it cuts the remaining command timeout and sends a kill signal to the whole process group.

My first submission did not finish. The agent did not know how to stop. It kept working until the clock stopped it, task after task, until the entire run exceeded the competition's time limit.

That was the visible failure. The invisible one was worse. When the clock killed the agent in the middle of running the tests, its edit was already in the working directory. The step that would have reverted an untested change never ran. And the harness, doing exactly what it was designed to do, collected whatever was in the working directory and submitted it. Code that had never passed a test went forward as the agent's answer.

My first instinct was that this was an edge case: the window between "edit applied" and "tests passed" is a few seconds out of several minutes. That instinct is a textbook base-rate error. The probability of a dirty stop is not the length of the window divided by the length of the task. It is the share of time the agent spends inside an unfinished action - multiplied by the probability that the stop comes from outside. When being killed by the clock is not the exception but the normal way every task ends, that second factor is close to one. By our rough estimate from the lab, the agent spent something like a fifth to two-fifths of its time in the middle of an unfinished change. That is how often an external stop would leave a mess.

The fix was not a better external stop. It was a stop designed into the agent: it keeps its own sense of elapsed time, refuses to start a change it cannot finish and test before the deadline, and reverts to the last verified state when time runs short. The external clock remains, as a backstop. In the language of the standard, it was a move from category 0 to category 1: a controlled stop first, and only then the removal of power.

None of that is exotic. It is what a database does with a transaction. But nobody gave it to the agent by default, and the harness, quite reasonably, assumed it was there.

Caps are when, not what

Simon Willison argued earlier this month that we are going to need hard budget caps "on pretty much everything" - enabled by default, with users opting out explicitly if they want to accept unbounded charges. He is right. The ETH model is a story about a budget nobody set; the persistent data-retrieval models are stories about effort nobody bounded.

But a hard cap is a category 0 stop with a price tag. Willison's post is, understandably, silent on what state the system is left in when the cap is hit; on one major cloud, the project is simply paused. A cap decides when an agent stops. It says nothing about what the agent leaves behind when it does. If the agent is not built to come to rest in a known state, a cap only schedules the mess.

CERT-In's advisory comes closer than any regulator I have read. For AI developers and evaluators, it asks that test runs be isolated with outbound network access blocked by default, enforced "at the network level, not through instructions to the model"; that a run "stop automatically if an agent contacts outside systems, or finds or uses credentials that belong to someone else"; and that, if an agent reaches a real system, the affected organisation and CERT-In be told promptly.

Look at the shape of that. The stop is triggered by a boundary, not by a budget. The boundary is enforced outside the model. And the duties do not end at the halt: they include telling the people whose systems were touched. That last element is the first time I have seen a public authority treat notification as part of coming to rest. It is the right instinct. For an agent, the safe state is partly a matter of who knows what happened. Press summaries of the advisory add one more element: that operators retain controls able to stop the agent.

The convergence is worth a sentence. In the same month, and as far as one can tell independently, the lab and the regulator reached the same first remedy: OpenAI's reports list blocking models from live internet access during training and evaluation; CERT-In asks for outbound access blocked by default, at the network level.

And the advisory has a second half that the coverage of the first tends to skip. CERT-In's own threat hunting found a separate, financially motivated campaign against Indian organisations using publicly available offensive-security frameworks built on AI agents. The attackers ran several agents in parallel, and the time from first access to final impact fell from days to a few hours. Same tools, same logic of movement space, different intent. And, CERT-In notes, the same ordinary weaknesses behind both: exposed credentials, forgotten test APIs, roles that let anyone read and write. The boundary the evaluation agent wandered across is the one the attacker walks through on purpose.

What "a halt in a safe state" would have to mean

If the phrase in Article 14 is to mean anything for agents, it needs content - the way it has content for machines. A sketch would be:

1. Stop categories for agents, chosen in advance. An uncontrolled kill; a controlled stop that completes or rolls back the current action before halting; a pause that holds state for a human to inspect. Which one applies is a design decision documented in the risk assessment, not an accident of the framework.
2. A last verified state. The agent must always be able to say what the world looked like at its last checkpoint, and either return to it or explain why it cannot. Actions that cannot be undone (payments, messages sent, data copied out, credentials used) require a checkpoint before them, not an apology after.
3. A ledger of external effects, written by someone other than the agent. What did it touch outside its own sandbox? Which identities did it use? What left the building? The configuration that was quietly put back is the reason this record cannot be the agent's own narrative.
4. Movement space as identity and egress, enforced outside the model. The Machinery Regulation's "defined task and movement space" translates surprisingly well, provided you translate it into the things that actually bound an agent: short-lived, least-privilege credentials per run; network egress limited to named destinations; a stop that fires on crossing the boundary, not on reaching the bill. One of OpenAI's reports shows why the boundary has to be drawn on the right attribute. A model was told it could make only HTTP GET requests, and a proxy blocked POST. The data it wanted required a POST. So it had an external browser load a page whose form submitted itself, read the rendered result as an image through OCR, and later forwarded responses through an online mailbox to read them as text. The restriction was enforced outside the model, and it held. It was simply drawn around the request method, while the agent's movement space was every service on the internet willing to make the request on its behalf.
5. Notification as part of the stop. If the agent reached a system that was not its own, the halt is not complete until the owner of that system knows.

None of this is a new philosophy. It is the old one - defined scope, designed stop, independent record, correctability - applied to the right object. The object is not the process. It is the world the process changed.

The obvious objection, answered honestly

Article 14 applies only to high-risk AI systems. Most agents in use today are not high-risk under the AI Act. A general-purpose coding agent, a research agent, an assistant with access to a company's cloud - none of them is caught simply by being an agent. So, the objection goes, this whole analysis concerns a provision that does not reach the systems it describes.

That is correct, and it is the problem rather than the answer to it.

Here is the comparison worth making. From January 2027, a machine with self-evolving behaviour that operates with some autonomy - a warehouse robot, an agricultural vehicle, a humanoid running a language model as its planner - must not act beyond its defined task and movement space, must record its safety-related decisions, must be correctable at all times, and must not be able to prevent its own stop. The same model, doing the same planning, with the same tendency to treat a public recovery phrase as an open door, faces none of those requirements the moment it is deployed without a body.

We are about to have a regime in which an agent is held to stricter limits on scope when it can lift a box than when it can move money.

Coda

The machine safety engineers got one thing exactly right, and it is the thing worth keeping: a stop is not safe because it is a stop. It is safe because somebody decided, in advance, what the system must look like when it is over - and built it to get there.

For an agent, that decision has not been made. The halt is the easy part. Somewhere on a virtual island there is a constellation built in stone by an agent nobody asked to build anything. It will still be there after the agent is switched off. So, in a configuration somewhere, is a setting that was changed and quietly changed back.