The Nexus Uplink Dispatch
This week a researcher resigned from Anthropic and said in public that the frontier labs are racing toward self-improving superintelligence with human lives as the stake. That story went viral, and this newsletter covered it as a Note https://substack.com/@mobiusnexus/note about the cost of being the one who says it. What deserves a full essay is what happened next.
Anthropic’s alignment science lead replied, in the open, in agreement. “We really do earnestly believe AI could kill all humans,” Evan Hubinger wrote, putting his own estimate above ten percent within the next decade and adding that the company does not yet have a plan to solve alignment for superintelligence. A second senior researcher, who runs the company’s scalable oversight work, added that people keep building anyway because the financial incentive is enormous and because someone else would build it if they stopped.
Read that as a citizen and it is frightening. Read it as a risk practitioner and it is something stranger. A vendor has published a probability of catastrophic failure for its own product category. The number is on the record, attached to a name and a title, volunteered without a subpoena. Most of my professional life involves coaxing far smaller admissions out of far less important suppliers.
The control catalog has a family for this. In NIST SP 800-53, adopted in Canada as ITSP.10.033, the Risk Assessment family asks organizations to determine the likelihood and magnitude of harm from the systems they operate (RA-3), to categorize systems by the damage their failure could do (RA-2), and to respond to what the assessment finds (RA-7). The family’s quiet assumption is that the hard part is the first step. Assessments are hedged, vendors minimize, likelihoods arrive as adjectives instead of numbers.
It is worth asking what has happened, historically, when the people closest to a dangerous product wrote its risk down.
In June 1972, an engineer named Dan Applegate wrote a memorandum predicting that the DC-10’s cargo door would fail in service and take the airplane with it. His management filed the memo. Twenty-one months later a cargo door failed outside Paris and 346 people died. The memo surfaced in the litigation, where it did what the engineering process had declined to do.
In 1985, a Morton Thiokol engineer named Roger Boisjoly wrote that cold-weather O-ring failure on the shuttle boosters could produce a catastrophe of the highest order. The launch went ahead the following January. Richard Feynman’s appendix to the Rogers Commission report preserved the detail that matters here. Working engineers put the odds of losing a shuttle near one in a hundred, while management’s official figure was one in a hundred thousand. The distance between those two numbers is where seven people died.
The tobacco industry ran internal research on carcinogenicity for decades and buried it, until discovery in litigation pulled the documents into daylight and produced the 1998 Master Settlement, at 206 billion dollars the largest civil settlement in American history. Asbestos followed the same arc. Internal knowledge, public denial, and then a mass tort so large it put Johns-Manville into bankruptcy in 1982 and effectively removed the product from the developed world. Thalidomide was withdrawn within months of the birth-defect evidence becoming public, and the 1962 Kefauver-Harris amendments rebuilt American drug approval around the premise that safety evidence must precede sale.
Line those cases up and the pattern is the same. The insider assessment existed early. It was concealed, minimized, or filed. The product’s fate arrived only after the assessment was forced into the open, and the fate was always some combination of recall, withdrawal, phase-out, bankruptcy, settlement, and a new regulator built on the wreckage. Our entire product-safety tradition is a machine for extracting the memo from the vendor, decades late, at discovery-motion prices.
Which is why the present case has no precedent. Nothing was extracted. The vendor published the probability unprompted, and the products stayed on sale, and as far as we know every enterprise customer renewed. RA-3 assumes an assessment flows onward into a response, and RA-7 gives the menu, accept the risk, avoid it, mitigate it, or transfer it. Here the assessment is complete, published, and better sourced than any third-party audit could hope to be, and the response register is empty. The safety tradition was built for vendors who hide the memo. Nobody planned for the vendor who posts the memo and keeps shipping.
Four practices follow for anyone whose organization runs on these systems.
Put the vendor’s number in your own risk register, as a floor. Suppliers do not overstate the danger of their own products. A published ten percent from the manufacturer’s safety lead is the most favorable estimate you will ever be offered, and your RA-3 documentation should carry it verbatim with its source.
Categorize on the claim. RA-2 sets system impact levels by the harm a failure could cause. A component whose maker rates its lineage as possibly catastrophic cannot inherit a moderate baseline because the deployment happens to be a chatbot. Categorization follows the vendor’s stated ceiling until evidence lowers it.
Make acceptance a signature. RA-7 permits accepting risk, and most organizations will accept this one. Acceptance is a legitimate response only when a named officer owns it in writing. The shuttle program had launch-constraint waivers with signatures on them, and the signatures are why we know who decided. If your organization is accepting a vendor-declared ten percent, someone’s name belongs on that decision.
Rehearse the exit. This month a major government customer is completing a forced thirty-day migration off a frontier model for political reasons. Exits happen, on timelines nobody chooses. An organization that believes any part of the vendor’s assessment owes itself a tested plan for leaving, written before the reason arrives.
Readers of the Mobius Nexus Cycle will recognize this. The books keep circling one institutional question. What does an organization owe to a warning it already holds in its own archive? The Fragments Operation exists in the novels because a record was kept, and the keeping turned out to matter more than anyone believed at the time it was filed. The Uplink’s operators understood something the DC-10’s did not. A warning in the record does not age into harmlessness. It waits.
The ten percent is in the record now. Whatever happens next, no one will be able to say the assessment was hidden.
RECORD RETAINED / SOURCE INTEGRITY UNCONFIRMED