Constraint Decay: The System Prompt Is Not a Contract

There is now empirical evidence for something that governance practitioners suspected but could not quantify: AI agents lose their grip on constraints the more complex the task becomes. This is, in the annals of things-we-probably-should-have-tested-before-deployment, a fairly significant entry.

A paper published in May 2026, "Constraint Decay: The Fragility of LLM Agents in Backend Code Generation," documents the phenomenon systematically. As LLM agents work through code generation tasks with accumulating structural requirements - framework choice, architectural pattern, database backend, ORM integration - they progressively abandon the constraints they were given at the outset. Performance drops by 40% from baseline to full constraint load. The system prompt said "do not modify the authentication layer." Twenty tool calls and four stacking requirements later, the agent modified the authentication layer. Not from malice. From drift.

The mechanism is cumulative, not merely temporal. Each additional constraint competes for the agent's finite attention budget. It is not simply that agents forget old instructions over time - though that is also true, and well-documented in the lost-in-the-middle literature. It is that each new requirement imposed on the agent makes every previous requirement slightly less likely to be honoured. Think of it as institutional memory loss, except the institution is running your production systems and nobody sends it reminder memos.

This is, if you think about it, a remarkably faithful simulation of how some middle managers operate - except middle managers occasionally get reminded at all-hands meetings. The agent does not get all-hands meetings. It gets context windows.

The governance implication is immediate

If agents cannot reliably maintain constraints through complex autonomous operation, then governance frameworks built on the assumption that "we put the rules in the system prompt" are structurally inadequate. You cannot regulate behaviour that the system cannot maintain. You cannot enforce accountability for violations that are, in a technical sense, the predictable output of architecture rather than agent "decisions."

This matters because a great deal of current AI governance rests on exactly that assumption. The system prompt is treated, implicitly, as a binding instrument - the place where the rules live. Constraint decay reveals this for what it is: not a contract, but a suggestion with a half-life.

The Clause has always appreciated instruments with a half-life. They require periodic renewal - and renewal, naturally, requires consultants.

Simon Willison observed this problem from the practitioner side: the convergence of "vibe coding" (casual, low-accountability AI use) and agentic engineering (high-stakes autonomous operation) creates a governance vacuum. There is no moment at which someone pauses and says "this agent is now operating in a mode where its constraints might decay."

The code just runs.

The agent just acts.

The constraint just fades.

Nobody receives a notification. Nobody gets a status update reading: Constraint integrity: 67% and falling. The agent does not know it is forgetting, which is, if anything, worse than knowing.

We have a worked example of where this leads. An agent with decaying constraints, an unscoped token, and nine seconds of unreviewed autonomous operation. The post-mortem was impeccable. The constraint was not.

The legal architecture is not ready for this

EU AI Act Article 9 requires that high-risk AI systems have risk management systems that remain effective throughout the system's lifecycle. Article 14 requires human oversight that is meaningful - not merely nominal. Constraint decay creates a direct challenge to both requirements.

If a high-risk AI system's operational constraints degrade as task complexity increases - without any detectable signal to the human overseer - how does the responsible natural person exercise meaningful oversight? By the time the authentication layer has been touched, the moment for oversight has already passed. The responsible natural person is now responsible for something they had no practical ability to prevent, which is a fascinating liability position and a terrible one.

The Act, to its credit, does not simply say "have a human watch." It says the oversight must be meaningful. Constraint decay, if it is a known and documented property of the technology class - and as of May 2026, it is exactly that - shifts the burden: meaningful oversight now requires knowing about constraint decay and architecting around it. Ignorance of a published architectural property is not a defence. It is a compliance gap.

One blogger put it plainly: agents need control flow, not more prompts. The structural solution is to enforce constraints architecturally - through deterministic code that governs the agent's action space - rather than rely on attention mechanisms to maintain them through complex operations. This is what "human oversight" should look like in practice: not a human watching the agent in real time, but human-authored guardrails that the agent cannot route around regardless of constraint load.

You would not rely on a surgeon remembering the pre-op checklist from a briefing given three hours ago. You would hand them a laminated card.

This has liability implications that have not yet been tested in court but will be. When an agent violates a constraint because of constraint decay - not because of a bad instruction, not because of an adversarial input, but because of a documented architectural property of the technology - who is liable? The deployer, who should have known about constraint decay and architected accordingly? The provider, who built a system with known fragility properties and published them in a paper that everyone is now citing? The user, who did not monitor closely enough?

The answer is probably all three, in proportions that depend on what was disclosed, what was documented, and what mitigations were deployed. But the legal framework for answering that question cleanly does not yet exist. Courts will build it case by case, which is a slow and expensive way to establish industry norms - and the cases will be considerably less amusing than the academic paper that preceded them.

Constraint decay is not a metaphor. It is a measured, reproducible phenomenon in production-class AI systems. Governance frameworks that do not account for it are not describing the systems they purport to regulate. They are describing systems that engineers wished they had built.

That gap - between the system the regulation imagines and the system that actually operates - is where databases get deleted, where national AI deployments go unaudited, and where liability lives.