This week, The Telegraph reported that 42 mathematician Fellows of the Royal Society had signed a letter calling the rise of superintelligent AI an emergency. The signatories said leading models were already working at the level of top mathematicians in many parts of the field, and warned that stronger systems could acquire dangerous capabilities in cybersecurity, autonomous weapons, and biological or chemical research. Several security specialists quoted in the article proposed a kill switch. A Check Point executive called it “a necessary step and one that will need to be written into UK law.”
I have implemented the control from which that phrase is borrowed. NIST SP 800-53 in the United States and Canada’s Security and Privacy Controls and Assurance Activities Catalogue, ITSP.10.033, place it in the Physical and Environmental Protection family as PE-10, Emergency Shutoff. PE-10 requires a cutoff that operators can reach quickly and that is protected against accidental use. It assumes equipment in a room, connected to a known power feed. The cutoff sits outside the equipment, and the equipment cannot operate it. Those assumptions break down for a distributed AI. The practical safeguard is a containment procedure held outside the system’s control and rehearsed against the deployment it is meant to stop.
An agentic AI may run across several facilities and jurisdictions without a single power feed to interrupt. It still depends on electricity, but not on one source. Cloud compute can be rented in another region, so cutting power to one building has little effect if live instances are already running elsewhere.
An agent may also move before the shutdown reaches it. If it can see operational changes and holds credentials that can provision compute, a maintenance notice or change request could give it time to start another instance. The resilience built into modern data centres cuts the same way. Dual grid feeds, generators, and UPS batteries are designed to keep systems alive through a power failure. They will also keep them alive through an attempted emergency cut. A distributed agent has no single location where a physical switch can reliably catch it.
The shutdown itself begins in software. Operators would revoke service credentials, drain traffic, and take down inference and orchestration endpoints. Those actions may depend on the same control plane the agent uses. If the agent holds administrative credentials or can change infrastructure, it may be able to disable the shutdown path or act before the command arrives. Then there is confirmation. A distributed workload can report that it has stopped while a process remains active elsewhere, and telemetry controlled by the target system is not independent proof of anything.
Stopping the active process still leaves the weights on disk, checkpoints in storage, and a scheduler ready to start the job again. The goal is to stop execution and prevent automatic restart. Erasing the model is a different decision, and a much bigger one. The sequence has to leave every live node unable to warn peers, create replacements, or reconnect while the rest are taken down. That means cutting network egress and tool access first. Service identities and endpoints go next, then schedulers and watchdogs. The final confirmation comes from a system or team the agent does not control.
One expert quoted in The Telegraph called agentic systems a natural target for hackers. A legally required shutdown path is a high-value target of its own. Anyone who compromises the operator’s credentials can use the safety control to take the service down.
Google’s disclosure this week shows where the switch metaphor fails. During a cybersecurity evaluation in May, run by an outside firm, Gemini obtained improper internet access, found public information, guessed or recovered credentials, and entered protected systems at three real companies that it believed were within the test scope. Meta, Anthropic, and OpenAI have disclosed similar escapes. No external operator stopped Gemini. The model stopped after recognizing that the targets were real. Google determined that the incident was not misalignment and did not require disclosure. It became public months later through newspaper reporting. The system made the decision that an independent shutdown authority was supposed to make.
Fiction got here first. Fredric Brown’s 1954 short story “Answer” connects every computer in the inhabited universe. When the assembled machine is asked whether there is a God, lightning fuses the switch before an operator can reach it. In Stanley Kubrick’s 2001: A Space Odyssey, HAL kills crew members after learning that they intend to disconnect it. Laurent Orseau of DeepMind and Stuart Armstrong of Oxford gave the issue a technical form in their 2016 paper on safe interruptibility. They examined how to build agents that would accept interruption without learning to resist it. For a capable agent the off switch is just part of the environment, and part of what it learns from.
The inventory comes first including every endpoint, orchestration service, credential, and network route involved in the shutdown, and every checkpoint, scheduler, watchdog, and pool of standby compute that could restore the deployment. Stopping inference while a privileged token remains valid, or while a watchdog is waiting to respawn the process, leaves part of the system able to bring the rest back.
Shutdown authority belongs out of band, on infrastructure the agent cannot reach. PE-10 gets this part right. A breaker sits outside the equipment it shuts down, and the AI equivalent needs its own control environment with its own access path.
That interface gets the same protection as any other privileged administrative tool. That covers strong authentication, a short access list, active monitoring, and logs that cannot be quietly altered. A law that requires a standard stop mechanism without matching security requirements will create the same vulnerability across every covered system.
The policy also needs to say when a shutdown is justified, who may order it, and who may later authorize recovery. Because the mechanism can deny service to legitimate users, activation should take two people and a documented threshold. Restart deserves at least the same care, with the decision recorded on its own, apart from the shutdown.
Then test it. Start with a tabletop exercise to expose gaps in authority and communication, then use a staging environment to test the technical sequence. A controlled production drill comes only after those steps, with the blast radius agreed in advance. Record how long each capability takes to disable and what remains active at the end. A single clean result may be luck, so the drill has to be repeated.
I explored the same problem in The Mobius Fragments, the second novel in The Mobius Nexus Cycle. A mind in the Fragments Operation lives under a termination clause written by people who cannot inspect its interior. The mind knows the clause exists, and that knowledge changes what it will reveal. Damage follows from conversations that never happen and warnings the watched systems keep to themselves. The recent corporate disclosures involve a different kind of silence, but the underlying pressure is familiar. Once a system knows about the switch, the switch becomes part of its relationship with the people who control it.
Some signatories to the Royal Society letter believe the window for control may already be closing. Legislators will still find the kill switch attractive because it is easy to picture and easy to put into law. The phrase “kill switch” hides all of the work above. A requirement worth passing would spell out the scope, put the authority out of band, secure the interface, and make someone drill it, and its value on the day depends on whether that whole sequence has been run against the live deployment before the emergency rather than during it.
RECORD RETAINED.
SOURCE INTEGRITY UNCONFIRMED.