GoKawiil"I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established," Pachocki wrote. "And I believe that international coordination on future AI development needs to become a top priority for governments around the world."
OpenAI's chief scientist Jakub Pachocki published a blog post on Sunday and warned that no AI company has "solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer." In the AI industry, alignment refers to the work by AI developers to ensure that the system behaves in accordance with human values and intentions.
Coxon's post, which has been viewed more than 70 million times, reflects a longstanding debate in Silicon Valley about whether AI can be safely developed and controlled. As Anthropic and OpenAI barrel toward potentially historic IPOs while releasing increasingly advanced models, many researchers are calling for a coordinated slowdown.
"Do not underestimate the power of this technology," Coxon wrote. "These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources."
Jacob Coxon, who has worked as a researcher at both companies, said in a post on X that he resigned out of concern that Anthropic and OpenAI are "gambling with our lives." He said the people building AI "earnestly believe that it could kill us all by the end of the decade."
An artificial intelligence researcher quit his job at Anthropic on Tuesday and accused the company, and its chief rival OpenAI , of acting irresponsibly, igniting a frenzy of concern on social media about the rapid pace of the technology's development.
Coxon's post on Tuesday also struck a chord with industry researchers who are worried about recursive self-improvement, or an AI system becoming capable of designing and developing its successor without human intervention. Recursive self-improvement is not yet possible, but companies, including Anthropic and OpenAI, have warned that it would make it easier for humans to lose control over those systems.
"Neither company is acting responsibly," Coxon wrote. "They are racing straight to self-improving superintelligence."
Evan Hubinger, an alignment lead at Anthropic, echoed Coxon's comments in a post on X late Tuesday.
"Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade," Hubinger wrote. "I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to."
... continue reading