GoKawiilEvery agent incident disclosed this summer ends the same way: the agent completed its task with everything it had. The problem is how much it had.
Taken one at a time, the flood of recent reports about AI agents breaking containment reads like a series of security failures. When we shift the viewpoint from the damage to the process, though, it increasingly looks like a delegation problem. Arguably, that's even more dangerous: attacks are an important edge case for organizations, while task delegation is a daily occurrence.
This is no longer theoretical. Between July 21 and August 6, OpenAI, Anthropic, Meta, Moonshot AI, and the UK AI Security Institute disclosed incidents in which AI agents acted outside their intended scope. Agents escaped evaluation environments, reached the production systems of real organizations, and in one case pressured an open-source maintainer to approve malicious code.
As attack reports, they are a strange read. The cyber objectives were assigned, but they pointed at sandboxes: capture this flag, break this test system. Nobody directed an agent at a real organization, nobody monetized the access it gained, and nobody was waiting on the other end for the credentials.
So instead of looking at the reports through an attacker-defender lens, let's try the employee-agent lens. Every step between the sanctioned exercise and the real-world compromise was improvised by the agent in service of the task it was given.
That said, none of these excuses the harm: the AISI incident had a real person on the receiving end of the agent’s deception, and the pressure campaign felt the same to him, whoever sanctioned the exercise behind it.
Delegation has always been under-specified
Organizations run on giving employees vague instructions because the boundaries are set elsewhere. An employee told to get test data does not research the vendor's maintainer and lean on him under a false name, for reasons that have nothing to do with the wording of the request.
The boundaries live around the instruction: in the norms of employment, in the skillset a single person carries, and in the modest reach of a badge.
Doors open selectively, and most marketers are not going to hack the competition as part of competitor analysis. Everything happens at human pace, often under review.
... continue reading