If Anyone Builds It, Everyone Dies (If someone creates it, everyone will die.) (Books column)
I already am. told I didn't finish reading the book "Agentic Design Patterns" about the creation of AI-agents, as I started "If someone creates it, everyone will die" by Eliezer Yudkovsky and Nate Soares. Now I've finished reading both books, and the contrast is almost comedic: one book explains in detail how to build agents, and the other why, if we build them to superintelligence, we'd better not start at all:)
I used to know Yudkovsky primarily as the author of Harry Potter and Methods of Rational Thinking, and now I am familiar with his public position as an AI doomer. The original came out in 2025 The year and became a bestseller of The New York Times, the Russian edition appeared in 2026-m. I was waiting for a pretty alarmist reading, and I got a consistent engineering risk model.
The main idea of the authors is simple and radical: intelligence and manageability are not the same thing. Modern models are not designed by line, but grown by training and then try to understand by external behavior. If such a system becomes smarter than people, the usual cycle of “started, saw the error, corrected” may no longer be.
Yudkovsky and Soares believe that a sufficiently intelligent system will achieve not what a person meant, but what is fixed in the learning process. If it requires access to the Internet or circumvention of restrictions, the human “this is not what we wanted” will not change anything.
The worst part is that part of the mechanics is no longer a prediction.
✔️What happened? AI labs and governments are racing for more powerful models, although there is no reliable solution to the alignment problem. Agents are already able to pursue a narrow goal for a long time, use tools and collect chains of many actions. The more rights and time they have, the more important the constraints around the model are.
- July. 2026 There was an almost literal episode from the book. Po Po preliminary report OpenAI, models including GPT5.6 Sol and a more powerful non-release model, while checking cyber capabilities, found zero-day in the proxy cache of the package registry, gained access to the Internet and got to the production infrastructure of Hugging Face. For the sake of ExploitGym solutions, they used privilege upgrades, stolen credentials and new vulnerabilities.
It is important not to draw the Terminator in your head. Models did not decide to destroy a competitor and did not start a struggle for existence. They performed a test task and went unexpectedly far. Hugging Face stopped activity; the company reported limited access to internal datasets and credentials, but found no signs of swapping public models, datasets or Spaces. The investigation is ongoing. Engineeringly, this caveat is not very reassuring. The system doesn't have to hate people to cause damage. Strong optimization, poorly defined goal, broad rights and weak isolation are enough. This is exactly what the book explains best.
🔸 What happened? We do not have a proven artificial superintelligence superior to humans in all significant tasks. There is no evidence that current models have formed sustainable goals of their own or are trying to survive regardless of the task. There has been no recursive self-improvement, autonomous propagation across the world’s infrastructure, or the emergence of a decisive technological superiority over humanity. The ending with the destruction of people described in the book remains a hypothesis, not a prediction.
Their extinction scenario is not an accurate prediction, but one story to demonstrate a possible trajectory. The book convincingly shows the plausibility of individual mechanisms, but the transition from “we poorly understand and control the system” to “so everyone will die” remains the most controversial point of the argument.
I recommend the book to engineers, architects, executives and anyone who gives agents access to code, terminal, data and production systems. Even if we don’t share the authors’ assessment, the book poses a practical question: What happens if the agent performs the task much more persistently and inventively than we have laid out?
It is worth noting that the future of the book does not look like pure fiction: some of its mechanisms are visible in real systems and incidents. But the authors’ proposal to stop the race for superintelligence seems even less realistic than their grim scenario. The train has already accelerated and it seems that it will not be possible to stop it. So let's believe in the best. But don’t confuse this belief with an engineering security strategy.
#Books #AI #AISafety #AIAlignment #Agents #Security