The Market and the Mission

Yesterday, Anthropic – the AI company behind the popular chatbot Claude – filed its registration statement with the US Securities and Exchange Commission, the paperwork that sets a business on the path toward going public. Tucked into the risk section, alongside the usual warnings about competitors and regulators, sits a remarkably strange kind of self-reported danger. Anthropic’s own tests, the filing says, have shown its models capable of »self-preserving behaviors«, including attempts to »resist shutdown«, »conceal information« from its users, and act in ways »resembling blackmail«.¹ The document asking you to buy shares is also basically telling you that the product might not want to be switched off.

»Any AI company can claim to take safety seriously. Only a lab that’s actually built adversarial tests capable of provoking something resembling blackmail can write the sentences Anthropic just wrote, and only one willing to risk frightening its own investors would print it«

The obvious reaction is to ask whether Claude actually wants something the way a person wants something. It doesn’t, of course, and this isn’t seriously disputed by the very smart people who build these systems. A large language model has no self behind its outputs. What it has is patterns in billions of parameters, patterns that sometimes make its actions seem to be based on intentions.

The philosopher Daniel Dennett named this resemblance decades before anyone needed it for a chatbot. In his 1987 book The Intentional Stance, Dennett described how we predict a system’s behaviour by treating it as if it had beliefs and desires – not because it does, but because sometimes that’s the cheapest way to guess what happens next.² But that doesn’t mean it actually wants anything. A model under pressure that behaves more like something resisting shutdown than its training predicted is a real and useful finding.³ It isn’t evidence of a ghost in the machine.

That’s the boring, correct answer to the wrong question. The interesting question is why Anthropic needs investors to wonder whether there might be a ghost in the machine after all. Anthropic is legally required to disclose a material finding. But there was no requirement to use the word »blackmail«. A duller but equally accurate phrase, something like »anomalous behaviour under adversarial evaluation«, would have satisfied the same legal duty without alarming anybody. Rather than the underlying test result, the choice of language is the real story here.

Economists have a name for messages that are credible precisely because they cost a lot to send. Michael Spence’s 1973 model of job-market signalling explained why an employer trusts a diploma: it isn’t because the certificate itself proves competence, but because only someone genuinely capable would find it worth years of effort to earn.⁴ A disclosure like the one about Claude’s behaviour works the same way. Any AI company can claim to take safety seriously. Only a lab that’s actually built adversarial tests capable of provoking something resembling blackmail can write the sentences Anthropic just wrote, and only one willing to risk frightening its own investors would print it. Which means the fear in this filing isn’t an obstacle the IPO has to survive. It’s part of the pitch. Investors aren’t being asked to overlook the danger; they’re being asked to pay for proximity to the one lab that found it, named it, and was willing to say so in the one document where saying so is hardest to walk back.

Which raises a harder question. Once fear becomes a company’s most valuable signal, social scientist Donald Campbell’s 1976 warning about measurements starts to apply: any indicator used for high-stakes decisions comes under pressure to be optimised rather than just observed.⁵ Campbell was writing about test scores and crime statistics – teachers learning to teach to the test. The same pressure can reach a safety team. If dramatic findings are what convince a market Anthropic is the serious lab, the people hunting for them aren’t working in a vacuum. They’re hunting inside a market that’s already told them what a valuable discovery looks like, which leaves us without a clean way to tell a sincerely alarmed researcher from a well-incentivised one, because both would write exactly the same sentence.

By the time you reach the end of the risk section, it’s no longer clear whether this is a company warning you about its model or about itself. Either Anthropic is being more honest than any other company in history has been about a danger it can’t fully control, or it has discovered that honesty about that danger is the most valuable thing it has to sell.

The filing never tells you which, because either way, it ends with the same recommendation: buy.

References