Please stop quantifying your AI existential risk vibes
Last week Evan Hubinger, who leads alignment science at Anthropic, said he thinks there’s a greater than 10% chance that AI kills all humans in the next decade.
Bryan Cantrill wrote a good response about how irresponsible it is for technologists to spread fear like that. He also made the point more directly on Bluesky:
Claiming a probability implies expertise
Most non-technical people (including journalists) hear “10%” from Anthropic’s “alignment science lead” and assume it’s an expert estimate. But nobody has observed an AI-caused extinction event, and there is no established field with a track record of predicting one.
During the COVID debates, epidemiologists had populations to study, previous outbreaks to compare, and well-tested biological mechanisms. AI extinction has nothing comparable. The people making these claims are often young computer programmers with narrow alignment expertise, not broad experience in computing, biology, infrastructure, government, or how complex systems fail.
Hubinger graduated from college in 2019. Coxon says he spent three years doing pretraining research at OpenAI and Anthropic. Each of them has less than a decade of professional experience.
These are not experts in the strong sense many people assume.
Claiming a probability implies evidence
Technical people often put probabilities on uncertain beliefs using Bayesian reasoning. That can be useful for comparing beliefs and updating them when new information arrives.
But these probabilities don’t come from actual data, a model, or observations. They are subjective credences built mostly from priors and judgment.
Warn people without making up numbers
Of course AI poses serious risks. Any new technology this powerful does, including the relatively weak AI systems we have today. It is reasonable to raise those concerns publicly, explain how things could go wrong, and warn people.
But publicly claiming that you believe there’s a greater than 10% chance that AI kills all humans in the next decade is wildly irresponsible when the number has no evidence behind it. Warning people does not require a made-up probability.
So please, if you work at one of these AI model companies, stop quantifying your AI existential risk vibes.