Anthropic co-founder calls for mandatory AI ‘kill switch’

Jack Clark, co-founder of Anthropic, told the BBC on Monday that governments should consider requiring AI companies to build independently verifiable "kill switches" into their systems, adding regulatory teeth to a growing wave of concern from inside the industry about the pace of artificial intelligence development.bbc

Clark said that while most major AI labs, including Anthropic, already maintain internal mechanisms for shutting down their software, voluntary safeguards may not be enough. "Should companies be required to have a kill switch? Is that kill switch subject to verification by a third party?" Clark said in the interview, calling these questions central to "the broader policy dialogue" around AI.bbc

A Week of Alarm

The remarks cap a turbulent stretch for Anthropic and the wider AI sector. On September 8, Anthropic researcher Evan Hubinger publicly stated that he believes there is a greater than 10 percent chance of human extinction from AI within the next decade, a figure Nobel laureate Geoffrey Hinton told the BBC was "not unreasonable". Those comments followed the resignation of researcher Jacob Coxon, who accused AI labs of "racing straight to self-improving superintelligence and gambling with our lives".bbc+2

On Saturday, Anthropic CEO Dario Amodei published a 3,800-word essay calling for a broad slowdown in AI capabilities development, arguing the industry needs "an additional one to two years" to implement protective measures. The essay drew swift endorsements from OpenAI CEO Sam Altman, Elon Musk, and Google Alphabet Inc. ↗ 0.78% DeepMind chair Demis Hassabis. President Donald Trump dismissed the warnings on Sunday, likening the critics to "very negative forces" and insisting the United States must maintain its lead in the technology.cbc+2

IPO Looming

Clark's push for regulation comes at a commercially charged moment for Anthropic, which is preparing for an initial public offering as soon as October that investors expect could value the company at roughly $2 trillion, according to the Financial Times — a figure that would make it the largest IPO in history. Investors told the FT that Anthropic's annualized revenue is on track to reach between $100 billion and $120 billion by the end of 2026, up more than tenfold over the course of the year.qz+1

Clark acknowledged the tension between calling for constraints and operating within a fiercely competitive market. He cautioned that allowing AI to remain a "totally unregulated industry" is a dangerous path. "We are rolling dice with immense risks," he said. "The point is, we need to alter the trajectory of this industry".bbc