GoKawiilis The Verge’s senior AI reporter. An AI beat reporter for more than five years, her work has also appeared in CNBC, MIT Technology Review, Wired UK, and other outlets.
When OpenAI CEO Sam Altman, Anthropic CEO Dario Amodei, Google DeepMind cofounder Demis Hassabis, and SpaceX head Elon Musk loosely agreed over the weekend to slow down AI development, skeptics spotted an ulterior motive immediately. The AI titans had declared that their aim was to “pace the frontier,” signing on at least partially to a proposal for embedding third-party auditors, regulating domestic labs, and reaching a global slowdown agreement. Their critics, however, argued they simply wanted to stop would-be competitors, kneecap the open-source movement, and avoid real legal safeguards — some dubbed it an outright “cartel.”
The truth is more complicated, according to sources across the industry. The three-step proposal, laid out in an essay by Amodei, is calling for changes long espoused by AI safety advocates. While it could become a substitute for regulation, under Trump, substantial regulation is unlikely anyway. But experts say that on an issue that’s only likely to grow in importance, AI leaders aren’t the best people to lead the charge.
“The industry as a whole needs new champions,” said Nick Reese, an adjunct professor at New York University and the Department of Homeland Security’s former director of emerging tech policy. “We’ve looked at people like Dario Amodei, Sam Altman, and Elon Musk as these people who are closest to the problem and working on it every day, and they’re the ones who know best. But the truth is there’s never been a realistic vision for what we’re building toward.”
“The industry as a whole needs new champions.”
Concerns about AI advancement have been escalating for months, sparked largely by revelations that swarms of agents were behind rogue hacks going on right under the noses of leading frontier labs Anthropic and OpenAI. Reports from both companies fueled the fire, as did the resignation of Anthropic researcher Jacob Coxon, who posted a public letter explaining his choice. “The people building AI earnestly believe that it could kill us all by the end of the decade,” he wrote, adding that neither OpenAI nor Anthropic is “acting responsibly” and rather “racing straight to self-improving superintelligence and gambling with our lives.”
Dire warnings about AI aren’t anything new inside the industry, but Coxon’s letter seemed to break containment. It’s been viewed more than 170 million times on X alone, while Coxon and his story have made the rounds in newspapers and on TV.
To a significant chunk of AI safety researchers and AI nonprofit workers, however, this is simply drawing more attention to an already widely discussed issue. “A lot of people are thinking of this as Dario’s idea, or it’s coming from the CEOs, but that’s false,” said Daniel Kokotajlo, an ex-OpenAI employee who now leads the AI Futures Project, an AI research nonprofit. “People outside the companies have been calling for this for years … There’s been this growing chorus of voices saying, ‘Please don’t build superintelligence soon. We are not ready. You need to slow down.’”
Kokotajlo said that after years of people “yelling at them to do this” — including more than 1,000 AI lab employees signing a public letter from July, which called for a slowdown in AI development after the OpenAI-Hugging Face incident — the CEOs are now “now bowing to that pressure and also claiming credit for it, [although] not rightfully.”
Overall, a handful of sources told The Verge they believe that the verbal agreement made by the AI leaders is a step in the right direction. NYU’s Reese said he “do[es] not think it’s all hot air.” Apollo Research CEO Marius Hobbhahn called it “a good idea,” saying “it’s one of the best things for safety in a long time if it actually happens.” The Midas Project’s Tyler Johnston said it was a “good sign.” Redwood Research CEO Buck Shlegeris said it was “some great news — obviously it’s hard to know whether this is going to convert into anything real, but I’m feeling cautiously optimistic.” All agreed, though, that there was still a lot of work to do to make this verbal commitment a reality, particularly when it comes to making it an ironclad agreement.
... continue reading