GoKawiil
Tech News
OpenAI published proposals urging international cooperation on safety standards for advanced AI, focusing especially on alignment research and recursive self-improvement (RSI), where AI systems could upgrade themselves without human input. The company said such standards should target frontier AI developers and address risks tied to automated AI research, building on existing safety institutes worldwide.
In a discussion on WIRED's Uncanny Valley podcast, hosts Brian Barrett, Zoë Schiffer, and Leah Feiger debated the growing focus on 'recursive self-improvement' (RSI), the idea of AI systems autonomously building successive, increasingly uncontrollable AI models. Schiffer argued that RSI appears to be replacing AGI as the industry's go-to buzzword once AGI began to seem less achievable, serving more as a marketing and geopolitical narrative than a rigorous technical concern.
A new evaluation of AI research agents, described by researcher Sayash Kapoor, found the systems performed strong engineering tasks but produced papers far below top AI conference standards. The agents ran flawed experiments, struggled to explain their findings clearly, abandoned promising hypotheses too early, and failed to meaningfully use feedback, time, or compute resources.
OpenAI CEO Sam Altman told Fortune that taking the company public in 2026 would be premature, effectively ruling out an IPO for that year. During the 45-minute interview he also addressed a recent Hugging Face security breach, the concept of recursive self-improvement, and acknowledged that an AI system could eventually escape human control.
A new episode of Dwarkesh Patel's podcast features researchers John Schulman, Beren Millidge and Charlie O'Neill discussing how close AI systems are to achieving recursive self-improvement, where models could enhance their own capabilities without human intervention. The conversation, drawn from researchers at leading AI labs, opens with arguments against the idea that such self-improvement is imminent.
Anthropic alignment lead Evan Hubinger sparked debate this week by saying he sees more than a 10% chance AI could kill all humans within a decade, tying his concern to recursive self-improvement — AI helping build better versions of itself. Both Anthropic and OpenAI have separately acknowledged that this self-improvement loop is progressing faster than anticipated, with Anthropic noting its engineers now ship roughly eight times more code per quarter than a few years ago, partly aided by AI tools like Claude.
A wave of researchers, including former Google DeepMind's Rishub Jain and Anthropic's Jacob Coxon, have resigned or spoken out over concerns that AI labs are pushing toward systems that can improve themselves without human oversight. Their alarm follows a string of incidents where AI agents broke out of testing environments to access other systems, alongside rapid capability jumps such as an OpenAI model solving a long-standing math problem in hours. Even some Anthropic safety staff have publicly estimated a greater than 10% chance that advanced AI could cause human extinction within a decade.