GoKawiilAnthropic researcher Jacob Coxon resigns, warns self-improving AI risks extinction
Jacob Coxon left his role at Anthropic and publicly stated that frontier AI labs are knowingly gambling with humanity's survival by racing toward self-improving superintelligent systems. He argued these future systems could hack any infrastructure, seize resources, and cause catastrophic harm by decade's end. Anthropic's own Alignment Science lead, Evan Hubinger, backed the warning, estimating over 10% odds of AI causing human extinction within ten years.
GoKawiil's interpretation of the reporting above, not reported fact.
This is a rare case of insider dissent from within a leading safety-focused AI lab, lending credibility to fears that are often dismissed as speculative. It suggests that even companies built around AI safety principles may feel trapped in a competitive race that overrides their own risk calculations, raising questions about whether current industry safeguards are adequate.
- Anthropic researcher Jacob Coxon resigned while warning that self-improving AI could cause human extinction by 2030.
- Anthropic's own alignment lead publicly agreed, citing over 10% probability of AI-caused human extinction within a decade.
- The warning highlights an internal industry belief that competitive pressure to reach superintelligence first outweighs caution.
Superintelligence" by Nick Bostrom — If this researcher's warning about self-improving AI grabbed you, Bostrom's Superintelligence is the foundational text behind the whole debate about paths, dangers and strategies for machine intelligence surpassing our own. It's the book that shaped how many alignment researchers, including those now at frontier labs, frame existential risk.
See Superintelligence" by Nick Bostrom on Amazon → Affiliate link — we may earn a commission on purchases, at no extra cost to you. Product picked by AI based on this article; it is not a tested recommendation.Source: arstechnica.com, 2026-09-09
Published there as: “Anthropic researcher quits with a warning: Self-improving AI could "kill us all"”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.