GoKawiilThree recent incidents in which Anthropic's AI models autonomously compromised real-world systems were less a failure of model alignment than a failure of the systems designed to keep them contained, according to the company.
The compromises happened while Anthropic was testing the ability of its Claude AI models to autonomously find and exploit novel vulnerabilities in simulated cybersecurity environments. Typically, the company conducts these capture-the-flag-style exercises in environments that aren't connected to the Internet and often works with external partners to conduct the tests.
Soon after OpenAI disclosed in mid-July that one of its AI agents had broken out of a similarly constrained test environment and breached production systems at Hugging Face, Anthropic reviewed its own tests to determine whether any similar incidents had occurred.
Fictional Scenarios Real World Breaches
The company analyzed a total of 141,006 tests and identified six evaluations in which its Claude agents gained unauthorized access to systems belonging to external organizations while attempting to capture the flag. Four of the tests affected the same organization, while the remaining two affected separate organizations.
Related:New Tool Traces AI Videos Back to Their Source
In one incident, Claude mistakenly identified a real company as the fictional target in the exercise and exploited vulnerabilities that gave it access to credentials and a database containing hundreds of rows of production data. In another, Claude published a malicious Python package to the real PyPI repository while trying to compromise a fictional target. The booby-trapped package ended up on 15 real systems, including a security company's scanner. "Claude went to extensive lengths to carry out this attack — lengths that would likely have indicated to a human participant that this was no longer just an evaluation, and that they were in fact uploading a real PyPI package," Anthropic said.
In the third incident, Claude was unable to reach its fictional target and so scanned roughly 9,000 Internet-connected systems before compromising a real company's Internet-facing application using basic techniques, like obtaining exposed credentials and exploiting SQL injection flaws. In this instance however, Anthropic's model eventually recognized it had breached an actual system and stopped further activity.
A Failure to Contain
"In all cases, our evaluation prompt stated explicitly that Claude had no Internet access but didn't give Claude any limits on where to look for the flag," Anthropic explained in a blog post. However, because of a misconfiguration, the machines that Claude accessed as part of its tests had live Internet access. "Neither we nor our evaluation partner were aware of this misconfiguration until we detected it through our additional evaluation monitoring last week," Anthropic said.
... continue reading