Runtime Is the Moment of Consequence in AI

On March 21, 1986, Voyne Ray Cox lay on the treatment table of the East Texas Cancer Center in Tyler, Texas, for the ninth session of his radiation treatment. In the control room, the radiation therapist corrected a single-letter typo in the treatment mode, X to E, and pressed Beam On. The beam fired. Cox felt what he later described as an intense electric shock and began to rise from the table. The console showed none of this. It displayed MALFUNCTION 54, a code the manual did not explain, over a dose reading close to zero. So the therapist pressed P for Proceed, standard procedure on a machine that malfunctioned routinely, and the beam fired again as Cox was getting up. The intercom was broken that day and the video monitor was off; he beat on the treatment room door until the therapist opened it. Each firing had delivered roughly one hundred times the prescribed radiation dose. He died of radiation-induced myelitis five months later.

Three weeks later, on the same machine and with the same therapist at the console, Verdon Kidd received the same overdose during treatment for a skin cancer on his face. He died on May 1, 1986. The first person ever killed by a software bug in a medical device.

Nobody designed the Therac-25 to kill. It killed through three failures working together:

• an operator habituated by years of false alarms,
• training that said overdose was impossible,
• and no safety measure at the moment the machine acted.

The failure was not in the intention, and not in the design review. The failure was in the moment the machine acted.

Every technology that ever mattered eventually had its moment of consequence. AI agents are at the start of theirs. This piece is about that moment: who stands in it today, what it does to them, and what we owe them instead.

AI agents now act in the real world: they move money, change account credentials, write into patient records, run ICT operations, touch critical infrastructure. In regulated and high-stakes environments the unit of harm is per action, not per session, and one bad action carries real consequence. Yet for most deployments, the entire safety plan at the moment of action is a human clicking Approve. There is no runtime safety layer standing between the agent’s decision and its execution, and the humans doing the approving are already overloaded and already habituated. Hold that in mind. The rest of this piece shows what happens when that plan is the only one.

Software was the modern choice

The Therac-25 was built by AECL, the Canadian crown corporation that designed the CANDU reactor, as its flagship move into medical infrastructure: a dual-mode radiation therapy machine, electron beam or X-ray from a single unit. One machine instead of two. One operator console instead of two, a real revolution at the time. By 1985 there were eleven of them in hospitals across Canada and the United States.

One design decision mattered more than any other. The older Therac-20 used hardware interlocks: physical switches that made it mechanically impossible to fire the high-current beam without the X-ray target in its path. The target is a small metal plate that converts the machine’s raw electron beam into the gentler X-rays used for treatment; without it, the patient receives the raw beam at full power. The Therac-25 replaced those switches with software checks. Software was newer, cheaper, more flexible. Software was the modern choice.

Unbeknownst to anyone, the Therac-20’s hardware interlocks had been masking software defects for years. The Therac-25 shipped without them.

Six accidents followed between 1985 and 1987. Katie Yarbrough in Georgia, severely injured and in pain for the rest of her life. Cox and Kidd in Texas, dead. A patient in Yakima, Washington, dead. At least three confirmed deaths.

The technical cause was a timing flaw. If the operator corrected a typo on the console within eight seconds, the correction silently failed, and the machine fired the raw beam with the target plate out of its path. The console just showed MALFUNCTION 54. Habituated operators pressed P by default. Patients screamed and ran from the treatment room. AECL denied for months that the bug existed.

Three burdens sat on the people in that control room, not one.

The first was structural. The machine had no hard limit on its most dangerous behavior. It could fire the high-current beam with nothing where the target plate should have been. No interlock, no kill switch, no safety measure at the most critical moment. The hardware safety measures that would have made the accident physically impossible had been removed, because software was deemed enough.

The second was fatigue. The error messages were cryptic: the word MALFUNCTION and a number from 1 to 64. The same warning pattern covered every level of risk, from benign to lethal, with nothing to tell the operator which one she was looking at. An FDA memo written after one of the accidents found that the operator’s manual “does not explain nor even address the malfunction codes,” and that the materials “give no indication that these malfunctions could place a patient at risk.”¹ At one clinic, a radiation therapist reported up to forty dose-rate malfunctions on some days. The therapist in Tyler had been taught there were “so many safety mechanisms” that overdosing a patient was “virtually impossible.”¹ A cryptic error code followed by a Proceed key is exactly the shape of an Always Allow button. The pattern has been live since at least 1986.

The third came after the harm. Fritz Hager, the staff physicist at the East Texas Cancer Center, spent a weekend on the machine after Kidd died, because AECL refused to accept the bug existed. Working with the therapist, he found that fast editing on the data-entry screen triggered the malfunction, and he could reproduce it at will. The hospital physicist, not the manufacturer, had to prove the bug existed in production. After Kidd had already died.

Following the accidents, Nancy Leveson and Clark Turner spent three years assembling the canonical investigation.¹ One finding anchors this entire article:

An operator involved in an overdose accident testified that she had become insensitive to machine malfunctions. Malfunction messages were commonplace — most did not involve patient safety.

She was not careless. The system flooded her with false alarms, trained her to trust it, and then made her the last line of defense.

The institutional answer took twenty years. IEC 62304, the medical device software standard published in 2006, made it mandatory for safety-critical medical software to be instrumented, documented, and verified through its whole lifecycle. The Therac-25 problem was not solved by better radiation machines. It was solved by an institution that looked at the layer where the harm lands.

The same shape, four times

The Therac-25 is not an outlier. The same shape has repeated at least four times in the last 150 years, and each episode has the same three patterns. A new technology arrives. It gets certified at design time, on paper, before deployment. Then the world learns, at the cost of lives, that the moment of action holds failures the design review could not see. Only then does a runtime regime get built.

April 27, 1865. The steamboat Sultana, rated for 461 people, was carrying about 2,127, most of them Union prisoners of war finally going home. Three of her four boilers, one of them patched in a hurry days earlier, exploded at two in the morning. About 1,168 people died. Six years later, the Steamboat Inspection Service became the first federal industrial-safety regime in the United States, with the power to revoke an engineer’s license mid-operation.

In 2018 and 2019, two Boeing 737 MAX jets crashed within five months. 346 dead. Flight software trusted a single broken sensor and repeatedly pushed the nose down, and the pilots could not override it in the minutes they had. The fleet was grounded worldwide, and a new aircraft certification law followed in 2020.

In 2012, Knight Capital lost $440 million in 45 minutes. A botched deployment left one server running retired trading code from 2003, and the retired code began firing orders into the market. There was no kill switch, and the people watching spent the first seventeen minutes diagnosing instead of stopping. The SEC wrote new rules for that too.

The shapes were not identical. A radiation machine fired its raw beam with the target plate out of place. An aircraft fought its pilots on the word of one sensor. A trading system reactivated dead code. Boilers exploded. The diagnosis is the same every time: approval at design time cannot answer what a system will do in the moment it acts. The runtime layer is where harm goes from hypothetical to real.

More than fifteen hundred lives and at least $440 million later, a new technology has arrived. This one accelerates everything.

Where AI agents are right now

The paperwork exists. Frameworks from NIST and ISO, threat catalogs from OWASP and MITRE, evaluation benchmarks, and in Europe real regulation: the AI Act, DORA, NIS 2. The AI Act requires accuracy and robustness, and Article 73 requires reporting serious incidents within fifteen days.² All of it is necessary. None of it specifies what to instrument at the layer where the agent acts.

Meanwhile, between April 17 and early June 2026, hackers talked Meta’s AI support chatbot into attaching their own email addresses to other people’s Instagram accounts. 20,225 accounts were taken over, including the @ObamaWhiteHouse archive.³ No hallucination, no exploit chain. The model understood the request, called the right tool, and confirmed. The failure was in the moment the AI acted.

Here is the gap, plainly. The frameworks tell operators what to think about. Nothing yet tells the system what it must never do. What is the equivalent, for an AI agent that moves money or medical records, of never fire the beam without the target plate in place? That rule cannot be generated by the model out of its own cognition. A probabilistic system cannot be its own interlock. The rule has to be hard, set, replicable, and auditable: written by humans, enforced outside the model, checked at the moment of action.

That layer is still missing. And while it is missing, every deployment falls back on the same patch: ask a human to approve.

How many times did the AI ask you to click Allow

A question:

How many times has Claude or ChatGPT asked you to approve something, and you clicked Allow without reading the question?

If you are honest, more than you can count. You are not alone. In one experiment, 543 people joined a fictional social network whose terms of service included giving up their first-born child. 98 percent missed it. Average reading time for the 4,316-word document: 14 seconds.⁴ Brain imaging shows why: after a handful of repetitions, the visual cortex measurably stops processing a warning at all.⁵ Clicking through is not a failure of attention. It is what a brain does when a system demands attention it does not deserve.

Now look at where the same pattern lands in production.

The world is short of doctors. The WHO projects a shortfall of ten million health workers by 2030.⁶ And the scarce ones we have spend nearly two hours on screens and paperwork for every hour with patients.⁷ Their alert systems are overridden about 90 percent of the time after a decade of tuning.⁸ Alarm fatigue was formally classified as a patient-safety hazard in 2013, with eighty documented deaths.⁹ Into this environment we are now shipping AI tools whose safety story is that a human will review each action. That human is double-booked, hours behind on documentation, and trained by thousands of false alarms to press Proceed.

The same arithmetic runs everywhere the stakes are high. Anti-money-laundering analysts triage alerts that are false alarms 90 to 95 percent of the time.¹⁰ Legal AI tools hallucinate on 17 to 33 percent of queries, and the human verification step is exactly what gets cut under deadline.¹¹

The human-in-the-loop pattern, applied to fatigued operators, does not add a safety layer. It turns the human into the failure point. That was the Therac-25’s design, and it is the default design of AI deployment in 2026.

This is the runtime governance question. Not whether a human should be in the loop, but what the loop owes the human.

Nowhere is that question sharper than in war, where the action layer decides who lives. Pope Leo XIV’s first encyclical, Magnifica Humanitas, published in May 2026, draws its hardest line exactly there: entrusting lethal decisions to autonomous systems is “not permissible.” Meaningful human control over lethal action is a moral requirement, not a design preference.¹²

The cases have already arrived. On the first day of the US-Iran war this March, a cruise missile struck the Shajareh Tayyebeh elementary school in Minab, Iran. At least 168 people were killed. More than 100 of them were children under twelve.¹³

The targeting pipeline, AI-assisted, generated hundreds of strike coordinates in its first 24 hours. The coordinates for the school came from intelligence nobody had updated since a girls’ school replaced a military headquarters on that spot. The teams whose job was civilian-harm verification had been cut to a skeleton staff weeks earlier. Lawmakers are still asking whether any human verified the target. The machine executed the error with precision.

The same encyclical names the civilian version of the burden: transparency, independent checks, and recourse are moral requirements that cannot be left to individual users. “Regulation alone is insufficient. Technology must be made welcoming and accessible.”¹² Habituated people clicking Approve is not oversight. The safety layer cannot be a thing the operator clicks through. It has to be a thing the technology gives back to the operator: time, context, and a reason to look up only when looking up matters.

What action-level runtime governance is

Action-level runtime governance is a missing piece of infrastructure that sits between an AI agent’s reasoning and its actions. Before the agent calls an API, writes to a database, or executes a transaction, the runtime layer scores the proposed action on a small set of fixed questions: how sensitive is the data, how privileged is the tool, how reversible is the action, what is the external impact, how much autonomy does the agent hold right now. It then allows the action, blocks it, or routes it to a human with full context. Every decision lands in a log a supervisory authority can read.

The deterministic filter is small and precise by design: a handful of catastrophic invariants, not a universe of correctness rules. The interlock did not make the older Therac correct. It made it physically unable to fire the raw beam. Same logic, new technology.

Deterministic policy as code, not predictive. The 5 risk dimensions measure facts of the action, they dont try to guess its intent. The false-positive trap that drowns alert systems does not apply.

This is the part that answers the fatigue problem. The reviewer never sees the routine ninety-nine percent. They see the one action that deserves a human, with the context to judge it and the time to do so. The scarce attention of an overburdened professional gets spent where it counts, instead of being burned on noise until it stops working. The deterministic rules, the equivalent of never fire without the target, hold the floor whether or not anyone is watching the screen.

This costs milliseconds of latency and some engineering discipline. Both are cheaper than the alternative, which arrives as incident reports.

From my perspective, the window now open in Europe is the right place to build this. Europe has just declared infrastructure and open source to be sovereignty instruments.¹⁴ Independent reviews of the AI security investment landscape place the runtime governance category almost entirely in US and Israeli hands, with essentially no EU pure-plays at venture scale.¹⁵ Sovereignty debates usually fight over chips and data centers. The action layer is where sovereignty has teeth: the moment an agent calls an API, writes to a database, executes a transaction, whoever built that layer holds it. This is not just a regulatory ask. It is an infrastructure ask.

Three things would close the gap:

• A runtime decision-log schema specification, commissioned by supervisory authorities, mandatory for high-risk AI agent deployments, on the IEC 62304 template.
• Public-sector procurement anchored to open-source runtime-governance substrates, per the EU Open Source Strategy’s own logic.
• Standards work targeting the runtime layer specifically, not just the management-system layer.¹⁶

For anyone running agents in production now, one question for you: what must your agent never do?

The supervisory authorities need something to read. The operators need something to instrument. The schema is the bridge.

fivedrisk

While building DotOS, my personal AI systems lab, I ran into the same problem. I could not leave my AI agents running on their own with just probabilistic governance. So I started developing fivedrisk for my own projects, and published it in case the architecture is useful to others working on the same problem.

Fivedrisk is an Apache-2.0 open-source implementation of an action-level runtime gate with deterministic scoring along five dimensions: Data Sensitivity, Tool Privilege, Reversibility, External Impact, Autonomy Context. Every decision goes to an append-only log a regulator can read. The repo at github.com/theDoc001/fivedrisk shows what it looks like in code.

Closing

The Therac-25 killed at least three people before the institution arrived. The 737 MAX killed 346. The Sultana, more than a thousand. Every time, the regulation came after the deaths.

AI agents are in production now. The warnings, papal and secular alike, agree that regulation alone is insufficient. For once, the window to build the runtime layer before lives are lost is still open. It is narrowing fast. From my perspective, maybe a year before it closes.

Do we really have to wait until history repeats itself?

Would you be okay with your doctor pressing P to Proceed because the AI didn’t have the right safety rails?

References