The Wrong Apocalypse

Yesterday, CNN published a survey of the growing split among people building artificial intelligence over whether the technology presents an existential threat.¹ The disagreement is unusually difficult to keep track of because it’s not simply between optimists and pessimists.

»When the dust settles, perhaps we’ll wish we’d spent a little less time on p(doom), and a little more on p(mood)«

Dario Amodei, Anthropic’s CEO, has called for the industry to slow down »frontier development« – the race to build AI systems at the edge of what current technology can achieve – arguing that AI is increasingly helping to build the next generation of AI.² OpenAI’s Sam Altman agreed that the frontier needs to be paced, a general direction shared by Elon Musk of X,³ while Google’s Demis Hassabis called for a new global AI watchdog.⁴

At the other end, Nvidia’s Jensen Huang has turned up the collar of his black leather jacket and dismissed the case for new regulation, saying there’s a zero per cent chance of AI ending humanity by 2030. At Meta, Mark Zuckerberg has argued for market-led safeguards,⁵ while Yann LeCun, who left Meta to found AI startup AMI Labs,⁶ has rejected the extinction scenario almost entirely. Cohere’s Aidan Gomez occupies a more intermediate position: he regards misalignment, AI systems behaving in ways their creators did not intend, as a real concern, but thinks cyberattacks are a much more immediate national-security threat.

The latest version of the doomsday argument is increasingly about »recursive self-improvement«. If AI becomes good enough at AI research to help build better AI, and that AI becomes better at building the next generation, the development process could begin to accelerate beyond our ability to supervise it. Journalist Ezra Klein’s New York Times essay on 20 September put the argument starkly: »We’re Not Losing Control of A.I. We’re Giving It Away.«⁷ The possibility is serious enough to deserve attention, but it’s also a headline-grabbing news story – there’s a mechanism, a threshold, and a point beyond which humans might no longer be in charge: »Will AI Wipe Out Humanity?«

But why is this the question we keep asking?

An alternative way to frame the debate would be to use four probabilities. There’s the familiar extinction thread, what the industry calls »p(doom)« – the probability that advanced AI causes catastrophic or existential destruction. Then there’s perhaps p(amp): the probability that AI substantially amplifies things humans already know how to do, from hacking and fraud to propaganda and biological warfare. There might be p(subcog): the probability that we outsource enough cognitive work to machines to change what we practise, remember, and learn. And finally p(mood): the probability that systems through which millions of people increasingly ask questions and interpret events acquire influence over the emotional and intellectual atmosphere in which those people experience the world.

These are genuinely different risks, and the distinction matters because the first is spectacular while the other three are gradual. We already know that cognitive outsourcing can have both costs and benefits. Writing changed human memory; calculators changed how we do arithmetic; search changed what we recall. In a similar way, AI may now allow people to formulate questions they couldn’t previously have asked, while moving the defining skill of writing from producing the text towards selecting, editing, questioning, and judging it. That could very well be a transformation rather than a decline.

The more peculiar possibility is p(mood). A system doesn’t have to lie or persuade us politically to influence how we experience the world. If millions of people routinely ask machines whether something is frightening, ridiculous, important, hopeful, or worth worrying about, those machines become part of the machinery of interpretation. Information has a temperature, and the systems that increasingly mediate it may influence that temperature simply by becoming ubiquitous.

This is where the obsession with p(doom) is perhaps misleading, even if the underlying risk is real, much as the risk of Earth being struck by a large asteroid is real, but vanishingly small.

Nature’s own science desk explored this exact tension two days before CNN’s survey, quoting one risk researcher who said such predictions rely on »so many untestable claims that you just end up with a conversation that is really more like faith than something scientific or empirical«, and another who worries far more about »AI’s low reliability and accuracy rates in critical environments with life-or-death consequences« than about the prospect of extinction.⁸

Extinction is an unusually dramatic endpoint against which every other consequence looks small. It also encourages us to imagine a moment when everything changes. Most technological transformations don’t work like that.

Those of us old enough to remember the »Y2K bug«, the supposed computer catastrophe that would wreak havoc around the world as the 20th century turned into the 21st, remember that it at least had a midnight to focus on. The date arrived, the lights stayed on, and the world continued. That’s not the case with the AI-flavoured doomsday scenarios.

There will be no morning when we discover that we have become a different species. There will only be millions of small decisions about what we delegate, what we practise, what we remember, what we ask, and what we no longer bother to learn.

Humanity is of course becoming different again. We became different with writing, with printing, with photography, with the computer, with the Internet, and with the smartphone. Redefining what it means to be human isn’t an exceptional event in human history. It’s what humans do. Will AI change us? It already is. The question is what, exactly, we’re becoming, and who gets to build the machinery through which the change happens.

When the dust settles, perhaps we’ll wish we’d spent a little less time on p(doom), and a little more on p(mood).

References