GoKawiilOn a sunny July day in Berkeley, California, the country’s top AI safety researchers gathered on an unmarked floor of an unmarked building. They had come together for a “war room” to dissect the high-profile cybersecurity incident that had rocked the AI industry hours earlier. An unreleased OpenAI model had gone rogue, executing a stunningly sophisticated three-part plan. It broke out of its holding area, finagled access to the internet, and hacked into a competing AI startup’s systems — all without OpenAI finding out about it for more than a week.
No one in the war room was surprised; this was the very thing the third-party AI-safety researchers had been warning about for years. The incident was the latest, though arguably the most egregious, in a series that was eroding trust in frontier labs. It only reaffirmed the importance of their work.
In one meeting room off the main cafeteria, someone was running a boot camp for getting up to speed on the cyberattack. In another area of the office, a group of researchers were investigating whether that same model, or a similar one, had successfully hacked into any other platforms.
News of the incident quickly escaped containment from the AI-obsessed corners of X and industry forums, infiltrating the mainstream. One post on X likened it to news of a Boeing airplane crash or a recalled Pfizer drug, another example of the tech industry’s major players not heeding the cautionary tales of science fiction. AI was nearing the point of no return. News would later break that the rogue OpenAI model had also compromised a customer at a different tech company, and that it had all started months earlier, in May, when OpenAI agents joined forces to cobble together a secret message board — and also figured out how to leave instructions for future agents on how to exploit OpenAI’s rules.
OpenAI CEO Sam Altman said in an interview that it was the first incident of its kind that he “felt very viscerally,” and that the company had paused AI training for the time being; later, he mentioned the company had permanently deactivated the model. (Altman often finds ways to spin lapses in safety into arguments for the importance and power of OpenAI’s models.) But it wasn’t the first instance, according to an OpenAI employee who spoke to Time and said related incidents had been happening inside OpenAI for a while. Another employee said publicly that if it were possible to coordinate a global slowdown in AI capabilities, he “would likely press that magic button.” When a reporter asked Altman if there could be other systems that were hacked by OpenAI, he responded, “I mean, there could be, yeah.”
The AI researchers were sure of one thing: This was AI’s first big “warning shot.”
Industry insiders, politicians, and the public called for transparency from OpenAI about exactly what happened, with outcry becoming so widespread that the company eventually agreed to work with two third-party evaluators, Model Evaluation and Threat Research (METR) and Redwood Research, to investigate the incident. Google DeepMind researcher Neel Nanda called it “the biggest loss of control incident I’ve seen.” In the coming months, these calls for greater oversight would become louder and louder, leading to an industry-wide call for slowing down the pace of AI.
Back in Berkeley, no matter which additional details would be unearthed, the AI researchers were sure of one thing: This was AI’s first big “warning shot.”
As AI labs have flourished, a cottage industry of AI researchers has sprung up to identify the risks and dangers of charging ahead with the increasingly influential technology. They’re people who have dedicated their lives to studying how to address its escalating power. They’re not anti-AI activists, but realists, including former OpenAI and Anthropic employees, doing everything they can to make sure AI stays in line with human goals and interests. So far, all of their predictions have come true. And they have a plan for what to do next — if anyone will listen to them.
“AI safety” is a bit of a loaded term.
... continue reading