GoKawiilOK, but this time it’s really scary
In his essay, Amodei primarily attributes this rapid change in public positioning on development speed to the OpenAI-Hugging Face incident, where a “swarm” of AI agents coordinated to hack into an outside entity without explicit instructions to do so. While the overall damage in that incident was minimal, Amodei said he worries that “a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage.”
Without a slowdown in frontier development, Amodei said he worries that, in six to 12 months, a similar AI agent swarm would be “capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage)…” That’s at least a somewhat more specific worry than the amorphous concerns that “AI could soon kill us all” publicized by some other AI researchers last week.
Any slowdown in the time it takes to get to that extra-capable, extra-dangerous model will give researchers crucial time to “greatly reduce the risk that something goes seriously wrong,” Amodei wrote.
Credit: Getty Images Anthropic CEO Dario Amodei speaks at a June 2026 event. Anthropic CEO Dario Amodei speaks at a June 2026 event. Credit: Getty Images
Amodei acknowledges that these kinds of public calls for a slowdown in AI development date back to at least 2023. At the same time, he says those earlier examinations of AI “alignment” (i.e. how an AI’s actions line up with its user’s and creator’s desires) were “like trying to study the psychology of humans by performing experiments on bacteria.”
The difference today, Amodei says, is the impending risk of recursive self-improvement (RSI) systems that can autonomously build better versions of themselves. While many researchers see this as a hard-to-define pipe dream, both Anthropic and OpenAI are now saying that recent trends point to this kind of RSI system coming together in the near future.