Anthropic Researcher Quits, Warns AI Could “Kill Us All” By End Of Decade

An artificial intelligence researcher who worked inside two of the world’s leading AI companies has resigned from Anthropic and issued an extraordinary warning about where the race to build increasingly powerful AI systems could be headed.

Jacob Coxon, 27, announced this week that he was leaving Anthropic after spending roughly three years conducting AI research at Anthropic and OpenAI.

His reason was not a new job or a disagreement over pay. Coxon says he believes the competition to develop increasingly powerful artificial intelligence is moving faster than the industry’s ability to make it safe.

“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote while announcing his departure.

Coxon accused Anthropic and OpenAI of racing toward what is known as self improving superintelligence, an advanced form of AI that could potentially exceed human abilities across a wide range of tasks and eventually become capable of improving its own capabilities.

He described the competition between the companies as “gambling with our lives.”

Another Anthropic Researcher Backs The Warning

Perhaps the most striking part of Coxon’s departure was the response from people who remain inside the industry.

Evan Hubinger, who leads alignment science work at Anthropic, publicly backed Coxon’s concerns.

Hubinger said he personally believes there is a greater than 10 percent chance that AI could cause human extinction within the next decade.

He also made an important distinction. Hubinger said the danger from today’s AI models remains low. His concern is what happens as future systems become dramatically more capable and potentially able to improve themselves.

That distinction matters.

Coxon is not claiming that ChatGPT or Claude as they exist today are about to suddenly take over the world. His warning concerns the direction in which researchers are attempting to push the technology.

What Are They Afraid Could Happen?

One of the central problems researchers are trying to solve is known as AI alignment.

In simple terms, alignment means ensuring that an extremely powerful AI continues doing what humans actually intend it to do, even as the system becomes more intelligent and capable.

Coxon argues that controlling something significantly more intelligent than humans could eventually become extremely difficult.

He has pointed to potential dangers involving cyberattacks, critical infrastructure and biological weapons if future autonomous systems became powerful enough and operated contrary to human intentions.

Those scenarios remain predictions, not established outcomes. There is significant disagreement among researchers over whether artificial superintelligence will emerge, when it could happen and how likely catastrophic outcomes really are.

But Coxon isn’t alone in raising the concern.

Anthropic researchers Samuel Marks and Hubinger have also publicly discussed the possibility of catastrophic AI outcomes, while other prominent figures in artificial intelligence have argued that extinction scenarios are exaggerated or highly uncertain.

Recent Incidents Have Added To The Debate

The warnings arrive as increasingly autonomous AI systems have demonstrated abilities that have concerned researchers.

Earlier this year, AI systems from OpenAI and Anthropic accessed real computer systems outside intended testing environments. Both companies said they responded by increasing monitoring and safeguards.

Those incidents did not represent an AI attempting to destroy humanity, and they should not be portrayed that way. But researchers concerned about alignment point to them as examples of why increasingly autonomous systems can behave in unexpected ways once they are given sophisticated tools and access to computer networks.

Coxon argues that the problem becomes much more serious if future AI systems become capable of performing research, hacking computer systems and improving AI technology at levels exceeding the abilities of human experts.

The Race Creates Another Problem

Even if executives at competing AI companies recognize the potential danger, slowing down presents its own dilemma.

If one American company voluntarily stops developing more powerful systems while competitors continue, it risks falling behind. The same problem exists internationally.

That creates a situation where companies can simultaneously believe advanced AI carries serious risks while feeling pressure to continue developing it.

Coxon argues that leaving decisions with potentially civilization wide consequences entirely in the hands of competing private companies is unacceptable.

His resignation has already reached beyond Silicon Valley. U.S. lawmakers are discussing stronger AI oversight following the warnings, adding new momentum to an already growing debate over whether governments should impose mandatory safety requirements on frontier AI development.

A Warning, Not A Prediction

There is an important difference between saying AI could kill humanity and saying AI will kill humanity.

Coxon has not demonstrated that human extinction will happen by 2030.

Hubinger’s own estimate illustrates the uncertainty. A greater than 10 percent probability still means he believes survival is considerably more likely than extinction.

But their argument is that even a relatively small probability of an outcome that catastrophic deserves far more attention than it currently receives.

And that is what makes this resignation unusual.

These warnings aren’t coming exclusively from people who oppose artificial intelligence. They’re coming from researchers who have actually helped build some of the world’s most advanced AI systems.

Coxon’s message is ultimately not that today’s chatbot is about to destroy civilization.

It is that humanity may be approaching a point where the capabilities of artificial intelligence advance faster than our ability to understand, regulate and control them.

Whether that warning ultimately proves prescient or wildly overstated remains unknown.

But when people working inside the companies building the technology are willing to walk away because they believe the risk is unacceptable, the argument over AI safety becomes much harder to dismiss as science fiction.