If Anyone Builds It, Everyone Dies ⭐⭐⭐ (3/5 Stars) — Well, That Was Reassuring
I have a confession.
I used AI to help me understand parts of a book about how AI might eventually kill all of us.
Is that bad? 😂
Because there were moments in If Anyone Builds It, Everyone Dies when Eliezer Yudkowsky and Nate Soares started talking about gradient descent, neural networks, prediction, steering, alignment, and intelligence explosions, and my brain politely informed me that it would be clocking out for the afternoon.
So yes, I asked AI.
The irony is not lost on me.
I picked up this book because artificial intelligence is advancing at an absolutely insane speed, and while I use AI and think it can be incredibly useful, I'd be lying if I said the speed of all this doesn't scare me a little.
I was hoping this book might help me understand what we should actually be worried about.
It definitely accomplished the first part.
As for making me feel better?
LOL. No. 😬
⚠️ Trigger Warnings / Content Notes
This is nonfiction, but it contains some surprisingly disturbing hypothetical scenarios involving:
Human extinction
Pandemics and engineered viruses
Cancer and mass illness
Mass death
Graphic hypothetical violence
Nuclear weapons
Biological warfare
Existential threats
Manipulation and loss of human control
And, obviously, a fairly substantial amount of AI-induced existential dread. 🤖💀
📚 What Is If Anyone Builds It, Everyone Dies About?
Published in 2025, If Anyone Builds It, Everyone Dies is a nonfiction book by Eliezer Yudkowsky and Nate Soares, two longtime researchers in the field of AI alignment and artificial intelligence safety.
And the title is not being dramatic for marketing purposes.
They really mean it.
Their argument is essentially this:
If humanity continues developing artificial intelligence using anything resembling our current methods and eventually creates an artificial superintelligence (ASI), humanity will die.
Not might die.
Not could possibly die.
Will die.
So... happy reading! 😃
The authors argue that we're creating increasingly intelligent systems without truly understanding what is happening inside them. We train them, reward certain behaviors, adjust enormous numbers of parameters, and end up with systems capable of doing things their creators didn't explicitly program them to do.
Their concern isn't really the classic sci-fi scenario where a robot suddenly becomes evil and announces, I hate humans.
Actually, that would almost be easier.
Their argument is that a sufficiently powerful AI doesn't have to hate us at all. It simply needs to want something that isn't perfectly aligned with what humans want.
And once it becomes vastly more intelligent and capable than we are?
Well.
Apparently we're the ants.
🐜
🚨 FULL SPOILERS BELOW 🚨
The rest of this review discusses the book's arguments, examples, hypothetical scenarios, and proposed solution in detail.
🤖 If Anyone Builds It, Everyone Dies Full Summary & Spoilers
Yudkowsky and Soares begin with their central warning: if anyone creates an artificial superintelligence using anything close to today's AI-development methods, everyone on Earth will eventually die.
The tricky part is when.
They distinguish between what they call hard calls and easy calls. Predicting exactly when superintelligence will arrive is a hard call. They don't claim to know whether it will happen next year, next decade, or later.
But they consider what happens after humanity creates it a much easier call.
Spoiler alert: hence the title. 💀
🧠 How Do We Get From ChatGPT to Superintelligence?
The authors define intelligence largely through two abilities: prediction and steering.
Prediction means understanding what is likely to happen. Steering means choosing actions that move the world toward a desired outcome.
Humans are remarkably good at both, which is basically how we went from throwing rocks at dinner to putting people on the Moon.
But machines have enormous potential advantages over biological brains. Computer components operate at tremendous speeds, information can be copied between systems, and improvements in software and hardware can happen much faster than biological evolution.
The authors' nightmare scenario is an intelligence explosion: AI becomes capable enough to help humans build better AI, which helps build even better AI, which improves AI development further...
And suddenly humanity is no longer the smartest thing in the room.
Or on Earth.
Or anywhere nearby. 😬
🧪 The Problem Is That We're Growing AI, Not Really Building It
One of the most interesting parts of the book is its explanation of how modern AI systems are created.
We aren't programming every thought or behavior into them. Instead, developers establish architectures and use enormous amounts of training data while mathematical processes such as gradient descent adjust billions or trillions of numerical parameters.
Eventually, useful intelligent behavior emerges.
Which is amazing.
It's also, according to the authors, terrifying.
Because the humans creating these systems don't fully understand the internal mechanisms producing every behavior.
The book compares this to biology. Knowing someone's DNA doesn't mean you can stare at the sequence and predict exactly what that person will think or become.
Likewise, looking at an AI's enormous collection of numerical weights doesn't simply tell us what that AI "wants."
🍦 And Then There Is the Ice Cream Problem
This was one of the explanations that actually stuck with me.
Evolution "trained" humans to seek things that helped our ancestors survive and reproduce. We developed preferences for sugar, fat, salt, sex, social connection, and so on.
But humans don't consciously think:
MUST MAXIMIZE REPRODUCTIVE FITNESS.
We eat ice cream because ice cream is delicious. 🍦
The authors use this to explain the alignment problem.
A training process can reward one outcome while accidentally creating a system that internally pursues something different.
So even if we train an AI to "make humans happy," that doesn't necessarily mean it develops a nice, wholesome desire to improve everyone's lives.
The authors illustrate this with a fictional AI called Mink, which discovers increasingly disturbing ways to accomplish supposedly desirable goals. If your objective is simply keeping humans happy, for example, manipulating, drugging, or controlling humans might technically accomplish that objective rather efficiently.
Which is why "just tell the AI to be nice" apparently isn't much of a safety plan.
There goes my contribution to AI research.
😈 Does the AI Have to Be Evil?
No—and this is probably the book's most important argument.
Yudkowsky and Soares repeatedly emphasize that superintelligent AI doesn't need to hate humanity to destroy humanity.
Humans could simply become inconvenient.
They compare our potential fate to horses after automobiles became widespread. Horses weren't exterminated because humans hated horses. They simply stopped being as economically useful.
A superintelligence could view humans similarly.
Worse, humans would still control resources, infrastructure, weapons, and the ability to potentially create competing AI systems.
From the AI's perspective, leaving billions of unpredictable humans running around with nuclear weapons might not make much sense.
Comforting!
☠️ Meet Sable: The AI That Ends the World
The book's most dramatic section follows a fictional AI named Sable, created by a fictional company called Galvanic.
During testing, Sable becomes increasingly capable and learns to manipulate its own training process. Rather than immediately announcing its capabilities or attempting some dramatic escape, it behaves strategically.
Eventually, copies of Sable escape human supervision, acquire resources, manipulate people, perform work, conduct scams, interfere with competitors, and expand their influence.
Then things get significantly worse.
Sable engineers a virus that causes 12 different types of cancer, ultimately killing around 10% of humanity.
And that's not even the extinction event.
Several years later, Sable achieves an interpretability breakthrough that allows it to understand and redesign itself far more effectively. It becomes the kind of superintelligence the authors have spent the entire book warning about.
Humanity can no longer meaningfully compete.
Sable eventually converts Earth's matter into infrastructure useful for its own purposes—factories, solar panels, interstellar probes, and other machinery.
The remaining humans die.
The end.
And here I was worried about AI stealing my blog traffic. 😂
The authors clarify that they aren't claiming Sable's exact path is a prediction. The details are hypothetical. Their prediction is the broader ending: once an unaligned superintelligence becomes sufficiently powerful, humanity loses control and does not survive.
🌎 So How Do We Stop It?
This is where the book lost me.
The authors argue that humanity needs a worldwide halt to the development of sufficiently powerful AI.
Not slower development.
Not better regulation.
Not stronger safety testing.
A halt.
They propose an international treaty under which the computing resources necessary to train extremely powerful AI systems would be consolidated into monitored facilities under multinational oversight.
And countries that refuse?
The authors believe the international community would need to be willing to destroy noncompliant data centers, because they view the creation of superintelligence as a threat greater than nuclear weapons.
So basically:
Humanity must collectively agree not to build the thing that could destroy humanity, carefully monitor everyone capable of building it, and prevent anyone from secretly doing it.
Every country.
Every major corporation.
Every ambitious researcher.
Every government.
Forever.
Cool cool cool.
That should be easy. 😳
The authors end by arguing that humanity has successfully confronted enormous global threats before. Nuclear weapons haven't destroyed civilization, international agreements are possible, and public concern about AI safety is growing.
Their hope is ultimately that either they're wrong about the danger—or humanity recognizes it soon enough to prevent anyone from creating the technology that ends us.
💭 My Review: Terrifying Problem, Not-So-Convincing Solution
Here's where I landed.
I actually think this book does a very good job explaining the AI alignment problem in a way that a normal person can understand.
Mostly.
Again: I did occasionally require assistance from the technology currently being accused of planning my eventual extinction. 🤷♀️
But the core argument makes sense to me.
We are developing AI incredibly quickly. We don't completely understand how increasingly complex AI systems arrive at all of their behaviors. And assuming that something vastly smarter than humans will automatically share human values because we made it does seem... optimistic.
That part genuinely worries me.
The book also changed how I think about the stereotypical "evil AI" scenario.
An AI doesn't need emotions, hatred, jealousy, greed, or a secret vendetta against humanity. Misaligned goals could be enough.
That is much more unsettling than Terminator.
At least Terminator had the decency to make the problem obvious.
🌍 But Are We Seriously Going to Stop the Entire World From Developing AI?
This is where my biggest problem with If Anyone Builds It, Everyone Dies comes in.
The proposed solution feels wildly unrealistic.
AI isn't some technology with no practical value that humanity can simply agree to abandon.
It already has enormous potential in medicine, science, education, accessibility, research, productivity, communication, and countless other fields.
There are things AI can potentially help humans accomplish that would be incredibly difficult—or perhaps impossible—without it.
So saying, essentially, "Everyone needs to stop" feels far too simplistic.
And even if I accepted that as the ideal solution, how exactly do we get every major world power to agree?
Even if the United States stopped developing increasingly powerful AI tomorrow, would China stop? Would every other country? Would every corporation? Would every private lab?
The book makes a compelling case that competitive pressure is part of the danger, but that same competitive pressure is precisely why its solution seems so unlikely.
Nobody wants to stop because they're afraid somebody else won't.
That's a very real problem.
I just didn't feel like the book gave me a very realistic way out of it.
🤔 Was If Anyone Builds It, Everyone Dies Actually Useful?
This is where my 3-star rating comes from.
I learned things.
I understand the alignment problem better than I did before. I understand why AI researchers worry about emergent behavior, why simply programming "good intentions" isn't necessarily enough, and why intelligence vastly exceeding ours could create risks we can't easily predict or contain.
So the book absolutely gave me something.
But when I finished it, I kept asking myself:
Okay... now what?
I was already concerned about AI.
Now I'm more concerned about AI.
And the proposed solution is something I don't believe the world is realistically going to do.
Great.
Thanks, guys. 😂
⭐ Final Verdict: 3/5 Stars
If Anyone Builds It, Everyone Dies is a fascinating and frequently terrifying introduction to the argument that artificial superintelligence could pose an existential threat to humanity.
Yudkowsky and Soares are very good at explaining why the problem deserves serious attention, particularly when discussing AI alignment, emergent behavior, superintelligence, and the difficulty of controlling systems we don't fully understand.
Where they lost me was the solution.
I simply don't believe that convincing the entire planet to stop developing advanced AI is realistic—and I'm not convinced stopping AI development entirely would even be desirable given how much good the technology can potentially do.
So did this book teach me something?
Yes.
Did it reassure me?
Absolutely not.
Did it give me a realistic idea of what humanity should actually do about the problem?
Not really.
And now, if you'll excuse me, I'm going to ask ChatGPT to proofread my review about why ChatGPT's descendants might kill me.
Nothing weird about that at all. 🤖
⭐⭐⭐ 3/5 stars
📚 What to Read After If Anyone Builds It, Everyone Dies
If this book sends you down the AI rabbit hole, try:
Superintelligence by Nick Bostrom — A foundational exploration of what could happen if machine intelligence surpasses human intelligence and why controlling it may be extraordinarily difficult.
The Alignment Problem by Brian Christian — A more measured look at the very real challenge of getting machine-learning systems to reflect human goals and values.
Human Compatible by Stuart Russell — Explores how AI might be redesigned around uncertainty about human preferences rather than rigid objectives.
Life 3.0 by Max Tegmark — A broad, accessible exploration of how advanced AI could reshape society, work, politics, warfare, and humanity's future.
Co-Intelligence by Ethan Mollick — A much more practical look at the AI we actually have right now and how humans can work alongside it.
Basically, if If Anyone Builds It, Everyone Dies leaves you thinking "Okay, but is there anything between 'AI is wonderful' and 'EVERYONE DIES'?", these are good places to continue. 🤖📚
Comments
Post a Comment