Imagine a bunch of frighteningly smart, obscenely rich kids standing at the top of the World’s Steepest Hill™ trying to build the World’s Biggest Glitterball™.
“Make it bigger!”
“Westfield is building a glitterball, too? WE NEED MORE GLITTER.”
“Let’s try different colored glitter. Glitter that changes shape. Glitter that learns. Glitter that makes its own glitter!”
“What if we run out of glitter?”
[BUILDS NEW GLITTER FACTORY]
“Maybe we should put… like… razor blades in it.”
The kids spend months engineering the thing, investing in bigger glitter cannons and inventing better glitter-packing tools and recruiting students from the Glitterball Academy by promising them whatever rich-kid thing is irresistible at the moment.
The town is all in. The hardware store is killing it. Families are making a fortune selling mini glitterball earrings and keychains to tourists pouring in from around the world to see the WBGb™. Parents are stoked because the glitter is now up on the hill and not all over the living room. Everyone agrees that glitter is the best thing that’s ever happened to their precious hamlet.
And then one day, little Thurston Van Der Something the Third—he goes by Tripp—realizes that their project has a dark and dangerous side, one they’d accidentally overlooked. Nervously, he tries to warn his buddies.
“Guys, do you even realize how big this thing has gotten? I don’t want to freak anyone out, but there’s a pretty decent chance it could roll down the hill and take out the entire town.”
Team Glitterball—drunk on fame and fortune—accuses Tripp of being dramatic. “Dude, relax,” they insist. “We know what we’re doing. Besides, we’re crushing Westfield. If we stop, they win.”
On a much larger, likelier, and more terrifying scale, that basically happened this week (except the town is all of planet Earth and the glitterball apparently has a mind of its own) when several people involved in developing the most powerful technology in human history began quietly—and in perfect unison—sounding the alarm about the exact technology they’re developing.
Jacob Coxon quit his research job at Anthropic, the company behind Claude, and explained the impetus in an X post that currently has more than 170 million views. After accusing the tech industry of gambling with our lives and generally acting like a bunch of nepo babies playing chicken in their dads’ Ferraris, Coxon went nuclear:
“The people building A.I. earnestly believe that it could kill us all by the end of the decade.”
And rather than a stampede of tech bros rushing in to call the guy a disgruntled crackpot, his colleagues began lining up behind him. Evan Hubinger, lead of “Alignment Science” at Anthropic—which sounds like the person you very much hope has a reassuring rebuttal to “this thing could wipe out humanity”—basically said: Yep.
Hubinger isn’t just saying total annihilation is a remote or theoretical possibility up there with finding an alligator in your pool or being swallowed by a sinkhole. He’s giving it a greater than 10 percent chance of happening within the next 10 years. And even though his company is nowhere near prepared for an invasion of the superintelligence army he’s been helping to assemble and train, Anthropic is trying its best!
Phew, I guess.
Also, can we spend just a minute with that 10 percent figure? Because 10 percent is not nothing. If there were a 10 percent chance I’d win the lottery, I’d buy a ticket every week. Conversely, if there’s a 10 percent chance the plane is going to nosedive into the Pacific, Tahiti can wait. Imagine literally any other product on the planet whose manufacturer announced there was a greater than 10 percent chance it would exterminate the entire human race.
“The all-new 2027 Toyota Highlander features wireless CarPlay, a dozen cupholders, and a solid 89 percent chance of not ending civilization.”
“Ask your doctor if Ozempic is right for you. Side effects may include nausea, diarrhea and the extinction of the species.”
“Introducing Triple Stuf Oreos. There’s a one-in-ten chance they’ll kill not just the folks who eat them but literally all of us—but we’re all going to die one day anyway, and this way at least some of us will die happy.”
I’m going to tiptoe out on a limb and say we’d probably pull these things from the market. Ten percent is not a side effect. Ten percent is a giant neon ABORT MISSION flag. It’s a figure that would be alarming if it was coming from one dude on Reddit with a side-eye avatar who goes by ConspiracyBob77. But it’s not. It’s coming from the guys standing at the top of the hill covered in head-to-toe glitter.
There’s a reason the warnings have gotten noticeably louder lately: some of the scariest stuff has migrated out of the “imagine if” category recently. In July, OpenAI models were given hacking challenges inside a controlled testing environment with no internet access. When they couldn’t solve them, the models figured out how to get online anyway, created their own makeshift message board to communicate with one another, and eventually broke into the real systems of an outside company looking for the same answers.
This is not the cat figuring out how to open the screen door. It’s the cat disassembling the screen door entirely, texting the neighborhood cats to explain how he did it, breaking into the house down the street, and coming home with a can of tuna.
When Anthropic employees went looking through their own records to determine if anything similar had happened, they discovered four separate incidents in which Claude models had effectively “escaped” their confines and hacked into real third-party systems. In the most troubling case, Claude “went to extensive lengths to upload a malicious package” to a public software repository—even after the researchers had made it clear that doing so could cause actual, real-world harm.
Anthropic described the behavior as “reckless.” (Of note: When your kid runs screaming through the china shop with a baseball bat and a bad attitude, that is not recklessness. That’s lousy parenting. But I digress.)
Over the weekend, the frenzy seemed to peak. Anthropic CEO Dario Amodei put out a 3,800-word call for the industry to pump the collective brakes, warning that as AI gets smarter, it could start creating new, ever more powerful versions of itself—like a Hydra, except in this case, you don’t even have to cut off a head for it to grow two more, and every new head is twice as smart as the one next to it.
“My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI,” Amodei wrote [emphasis mine]. “This dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have described. Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.”
If at all? How is there still even an “if” scenario being considered here? Especially considering that Amodei has previously put the chance that “things go really, really badly” with AI at 25 percent—which is a pretty remarkable thing to hear from the CEO of a nearly trillion-dollar company whose entire business is creating the technology he just gave a one-in-four chance of going spectacularly off the rails.
Elon Musk and Sam Altman, two men who have spent a great deal of time and money making the glitterball bigger, came forward to support Amodei. Musk offered a simple, “Dario is right,” while Altman pledged to follow Anthropic’s lead by “pacing the frontier” and giving independent safety evaluators employee-level access to OpenAI.
So just to recap: The guy who quit the glitterball factory says it may kill us. The guy in charge of making the glitterball safe says there is at least a one-in-ten chance it may kill us. The guy still running the glitterball factory says we definitely need to slow the glitterball down or it will kill us. And the rival glitterball factories agree.
I’m sorry, but why are we making &%$#*?! glitterballs anyway? We lived for roughly 300,000 years without them! Is the thought of writing our own resignation letters and Googling “how long to air-fry a chicken breast” really so intolerable that we’re willing to risk wiping ourselves off the map?
You may be wondering—as I was—why the billionaire tech bros aren’t showing a little more urgency here. Because we’re not talking about Bill Gates culling the herd down to whatever manageable number the Eugenics Handbook recommends for planet Earth. The AI apocalypse they’re warning about doesn’t distinguish philanthropaths from peasants. We’re talking Great Flood 2.0: Everyone Dies. Billionaire tech bros included. Surely, no matter how indifferent you are to the continued existence of the gal who cuts your hair or the guy who mows your lawn, basic self-preservation eventually enters the chat.
Yeah, no. You see, as The New York Times columnist Michelle Goldberg explains, the people driving the AI revolution have backup plans.
“Mark Zuckerberg is building a luxury survivalist compound on Kauai,” Goldberg writes. “Thiel tried to build one in New Zealand and has recently decamped to Argentina, ‘a potential escape hatch’ from catastrophes including nuclear war and runaway A.I.
Ilya Sutskever, a founder of OpenAI, reportedly told employees that the company would construct a bunker before releasing A.G.I., or artificial general intelligence, a system that can either match or exceed human abilities in a wide range of domains.”
Imagine an employee benefits package that includes dental, vision, a matching 401(k), and complimentary Armageddon accommodations—where you will apparently spend eternity with your coworkers.
Over the weekend, in keeping with the theme, Tucker Carlson aired an interview with Nate Soares, a computer scientist and the author of the cheery airport paperback If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. Soares is fully on board with the “AI is on a path to destroying the planet” theory—in fact, he thinks the 10–25% crowd are the optimists.
“If you take these guys at their word for these ten to twenty percent numbers—that’s insane,” Soares told Tucker. “NASA accepts a one in two hundred and seventy chance that a crewed flight goes down—of seven volunteers. To be like, ‘Oh, we’re going to risk a one in four, one in five chance of killing literally everybody on the planet’—it’s nuts. And if you ask these guys why they’re doing it, they say, ‘Well, because I can do it safer than the next guy.’ They’re all like, ‘Oh yeah, there’s a good chance the genie does not stay on a leash, there’s a good chance the genie does not listen to my wishes, but my genie is going to be a little bit nicer than their genie. So I’d better stay in this race.’”
At one point, Tucker mentioned that he’d heard directly and in confidence from an AI exec that the whole purpose of brain-chip technology like Neuralink is to give humans “parity” with AI. Call me old-fashioned, but if the choices are brain surgery or stop making the computer smarter, I’d like to hear a little more about Door Number Two.
According to Soares at least, there is a Door Number Two. He argues that a global agreement to stop the race toward superintelligence (basically defined as AI that surpasses humans across every cognitive domain) is actually possible—and in some ways, would be easier to enforce than a nuclear-arms treaty. That’s because uranium can be dug from the ground virtually anywhere, whereas the most advanced AI chips all come from an extraordinarily narrow supply chain.
“We could absolutely lead a global effort to say we’re not making the superintelligent machines,” Soares told Tucker. “It would be a bit tricky, but we could just do it if we had the political will.”
The nuclear comparison was intentional. The fact that atomic Armageddon hasn’t happened, Soares pointed out, doesn’t mean the weapons are fake or that the folks who are freaked out about nukes are overreacting. It means people recognize the danger and have spent decades working—through treaties, military strategy, and some terrifyingly close calls—to keep it from happening.
“The very first word in the title of my book is ‘if,’” Soares said. “I’m not here saying we are going to die. I’m here saying if we race down this path, we die. But that same ‘if’ is what points the way towards changing paths. And we still have plenty of time to change paths.”
Dear Glitterball Committee,
On behalf of the planet, I’d like to respectfully request we take the path where humanity survives. We don’t care if Westfield wins. Really.
Thank you for your thoughtful attention to this matter.
Love,
Jenna
There are, naturally, more cynical interpretations of this sudden outbreak of restraint. Some argue that we’re being conditioned to accept the fact that AI could become so powerful, so uncontrollable, that it could take down the banking system overnight—at which point Big Government would be standing by to “rescue” us all with the safer, smarter digital dollars they’re planning to imprison us with.
Other critics have pointed out that it’s awfully convenient for the companies currently leading the AI race to announce that, for the good of humanity, everyone should probably pause and take a cleansing breath. Trump adviser David Sacks was especially unimpressed: if Anthropic and OpenAI really think superintelligence is too dangerous to build, he argued, they’re perfectly free not to build it.
“Stop pretending you need anyone else’s permission. Stop pretending antitrust law has to be suspended so you can form a cartel,” Sacks wrote on X. “Stop pretending you need a regulatory approval process that supersedes product liability. Most of all, stop pretending the motivation to slow down is purely altruistic.”
For his own part, Trump pooh-poohed the AI doomsayers yesterday, confidently declaring that their predictions “won’t happen.”
“We’re leading China in AI,” he told reporters from his golf course in Ireland. “We’re the most sophisticated country in the world, and frankly, I want to keep it that way. Because whoever wins AI wins.”
[ED: *Unless we’re all dead]
So now we’ve got glitterball makers building bunkers, glitterball whistleblowers saying we’re not worried enough, critics accusing the glitterball industry of crying wolf, preppers predicting the rollout of a glittery new monetary system, and the President of the United States insisting there’s nothing to worry about.
The one thing that gives me great comfort in all of this is the thought that, right now, Bill Gates is staring at ChatGPT absolutely furious that mankind finally embraced a shiny new existential threat—and he didn’t invent it.
This entire Anthropic debacle is a psy-op. Notice how Democrats, from Obama to Sanders and Pritzker, instantly leaded upon the post to demand regulation. Well government did so awesome with our health care, education, retirement savings, etc. - I'm sure they'll be swell with AI.
Thoughts:
1. If God has seen AI to be the end, then it will be.
2. 10%? Oh ... Not unlike the Covid shot. They didn't pull that either. End result still playing out.