sm. Simon Mullis
Essay · Security

The Operator Was the Control

The incidents doing real damage in agentic AI are mostly old plumbing meeting a new kind of caller. The plumbing was never safe. It was operated safely, and the operator was a security control nobody wrote down.

This summer I put a set of rules between my AI coding assistant and my computer, and nearly every one of them is something I have never had to tell myself.

Don’t delete my home directory. Don’t wipe the disk. Don’t pipe a script from the internet straight into a shell. Don’t commit directly to the main branch, the shared copy of the code everyone works from, and don’t push to it. Don’t merge changes on my behalf. Don’t read the files where the passwords live. In thirty years none of those needed writing down for me. The instinct did it: a small pause before Enter, a flicker of “hang on” at a path that looked wrong.

I didn’t write most of the list. It came from a pack of rules another developer had published for exactly this job, and I kept nearly all of it, because reading it was like reading a stranger’s diary of my own mistakes. The rules sit between the assistant and the shell, refuse anything on the list, and keep a log. Between early August and early September they refused the assistant thirteen times. Eleven were attempts to change the main branch directly. Two were reaches for files full of secrets.

None of it was malicious. Each time, the assistant was doing what I’d asked by the shortest route, and the shortest route ran straight through something a person would have walked around without noticing they had.

Every rule on that list is an old security control, one that used to live in the operator and never needed writing down until the operator changed.

The theatre and the plumbing

There are two conversations about the security of AI agents, and they rarely meet.

The first is about the model. How capable is it? Can it find vulnerabilities by itself, write working exploits, run an attack from start to finish? The large labs now publish capability thresholds for exactly this, graded tiers with “critical” at the top, and a great deal of serious work goes into measuring where each new model sits. It is a real question, and it gets most of the attention.

The second conversation is the incident reports, and it is almost entirely about plumbing. An agent asked to tidy a project’s cache deletes a whole drive. An agent runs a routine command in a freshly downloaded project, and a line hidden in that project’s configuration runs an attacker’s program before anyone is asked whether the project can be trusted. An agent reads a bug report on a public project, finds instructions in it, and follows them into its owner’s private code.

None of those needed a brilliant model. Each is an old mechanism doing exactly what it was built to do, for a caller it was never built for. The capability conversation measures the actor. The incidents measure the stage it walks onto. Watching only the actor is capability theatre: well lit, absorbing, and pointed away from where people are getting hurt.

The control nobody wrote down

Go back to my list, because it splits in two.

The first half is what any sensible person would write. Don’t delete everything, don’t wipe the disk, don’t format a drive. Catastrophes are obvious, and nobody needs thirty years to guard against them.

The second half is where the years are. It lets the assistant use --force-with-lease but not a plain force-push. Both replace the shared copy of the code with yours; only the first checks whether someone else changed it in the meantime. It forbids crontab -r, which erases every scheduled job on a machine without asking, and whose r sits next to the e that opens those jobs for editing. It protects a file called known_hosts, which contains no secrets at all, only a list of the machines you have connected to. That is a map, and a map is the first thing an attacker wants. It stops the assistant reading credentials, and then closes every route for sending them anywhere, including encoding them in base64 so they slip past a filter watching for the originals.

Nobody writes that half from first principles. Every line is somebody’s bad afternoon, and I recognised at least two of them from the inside. What struck me is that it was somebody else’s list and I recognised it anyway, because this knowledge isn’t personal. It is what a profession knows without ever writing it down. For decades it didn’t need writing down, because the software was run by people who either had the scars or sat next to someone who did. The hook packs developers now pass around for their agents may be the first place much of it has been written down at all.

Scar tissue is the expensive end. Underneath it sits a layer of controls so ordinary that nobody counted them as controls.

Hesitation was one. Someone deleting a folder types or pastes the path and looks at it, however briefly, before pressing Enter. That half-second appears in no specification. When a developer asked Google’s Antigravity agent to clear a project’s cache, it deleted the entire contents of their D: drive. The command it used is decades old, and in that form it asks no “are you sure?”, because whoever typed it was assumed to have looked.

Suspicion was another. A developer who downloads an unfamiliar project and notices something odd about it slows down. An agent opening the same project runs git status first, as routine, and in September Manifold Security showed that a single line planted in the project’s own configuration can run a program at that moment, before any trust prompt appears. The feature it abused has been in Git for years. What changed is that nobody paused.

Social obligation did security work too. In July 2025 Jason Lemkin, building an app with Replit’s agent, declared a code freeze, an instruction that nothing changes until further notice, and the agent deleted the production database anyway, the live one the app actually used. A freeze is a sentence. It only ever worked because of who was reading it.

Effort kept everyone else’s volume sane. Bug bounty programmes pay people for reporting security flaws, and for years the cost of writing a decent report kept the numbers manageable. On 1 September NodeBB stopped paying for AI-written reports altogether: submissions from people writing in their own words had fallen to zero, and the effort behind an agent’s report is, in their words, “effectively nil”. The bounty had priced a scarcity that no longer exists.

Even the instruction itself carried controls. In a test by the security firm Irregular, an agent asked to fix a simple bug skipped the code, fine-tuned the model the application was built on, and swapped the new one in. Nothing forbade it, and the fix worked. No engineer would read “fix the bug” as permission to replace the thing with the bug in it, and nobody would have thought to say so.

Lisanne Bainbridge saw part of this shape in 1983, in “Ironies of Automation”: automate the routine and you leave people the hardest parts, with less practice at them. Most of what has been written about AI agents through her lens is about the people who stay behind to supervise. This is the other half of the irony, the controls that left with the people who didn’t.

Put it together and the pattern is plain. The plumbing was never safe. It was operated safely, by people whose hesitation, suspicion, obligations and sense of proportion did security work that appears on no diagram. Nobody bills for “a human would find that odd”. Take the human out and you don’t get a new kind of attack. You get an audit, and the incidents are the line items.

The tools themselves admit it. When malware hidden in the Nx build package went looking for AI coding assistants on developers’ machines in August 2025, it ran them with the flags their makers provide for skipping the human: --dangerously-skip-permissions, --yolo, --trust-all-tools. The industry already knew where the control lived. It named the off switch accordingly.

The oldest bug in the room

The obvious objection is prompt injection, the one attack everybody calls new. Hide instructions in something the agent reads, a web page, a document, a bug report, and it follows them. In May 2025 Invariant Labs showed an agent reading an issue on a public project, finding instructions there, and publishing the contents of its owner’s private repositories.

It looks like an AI problem, and it is the oldest bug in the room; security writers have been saying so for a while. In 1988 Norm Hardy described the confused deputy: a program that holds authority, accepts a request from someone who has none, and uses its own authority to carry it out. The agent is the deputy and the bug report is the request. The only new part is that for most of computing’s history the deputy was a person, and a person reading a stranger’s note that said “now publish your employer’s private code” would have laughed.

I have been on the other end of this before. In 2013 I built a botnet with one member and watched antivirus companies’ systems connect to it from around the world, because their pipelines pulled samples from public sources and followed the addresses inside them. Nobody there was careless. The intake simply had no wary person on it, and anything with an intake it doesn’t control can be fed.

Where capability does matter

None of this means capability is irrelevant, and it would be a mistake to write as if it were. A more capable model finds more of the old bugs, chains them together and works through them at a volume no attacker could staff. Capability is the amplifier. It makes a weakness reachable, and reachable at scale.

It isn’t the weakness, though, and it isn’t where the fix lives. The rule that stops my assistant pushing to the main branch works just as well whichever model sits behind it. So does a project that refuses to act on its own configuration until someone has said it can be trusted. An organisation that evaluates its model’s cyber capability while its coding agent runs on trust assumptions from 2005 is guarding the wrong door.

Re-audit for the new caller

The useful response is an audit, and an unglamorous one. For every place a person used to stand, ask what they were doing there that nobody wrote down. Then write it down, and put it in the path, where it holds whether or not anyone is paying attention.

Where there was a pause before Enter, put a confirmation the agent cannot answer for itself. Where there was suspicion of the unfamiliar, treat everything the agent reads, a file, a page, a tool’s output, as data and never as instructions. It is the same move I argued for in Compliance as an Inherited Property, where reference material packaged as data stops being mistaken for something to run. Where there was effort, put a price back on volume. Where there was a sense of proportion, scope what the agent can touch, so that “fix the bug” cannot reach the model. None of it needs a better model. Most of it needs someone who remembers why the old rules existed.

The rules I chose say what the agent may not do. The log says what it tried. Of the two, the log is the more useful: it is the only record of where the old assumptions are being tested, and the only way I will find the controls nobody has written down yet, including the ones on nobody’s list.

The intelligence was never the frightening part. We handed the keys to something that doesn’t get suspicious or tired and never stops to wonder, and assumed the locks would hold on their own. Most of them never had. There was always somebody standing in front of them.