GoKawiilArtificial intelligence agents merrily breaking free and hacking other systems might seem like a sign of the impending machine uprising. In reality, it happens when we push remarkably clever, but also kind of boneheaded, algorithms to follow our every command.
I was first alerted to this looming agentic AI cybersecurity shit show in late 2025. Dawn Song, a UC Berkeley professor and one of the world’s top experts on AI and cybersecurity, grabbed my arm as I was walking out of the academic conference NeurIPS. Song told me that I should warn people about the havoc likely to result from AI’s rapidly advancing hacking skills. She is hardly prone to AI hype, so I duly did.
But things have escalated rapidly, even in the last eight months. A string of incidents involving freewheeling AI agents that broke out of their confines and hacked into outside systems with abandon shows just how powerful this technology has become. I caught up with Song, who recently joined Meta, to ask where things might go next and what we ought to do about it.
The bad news is Song thinks AI hacks will get worse before they get better. The good news is it seems clear why these little rascals are going off the rails in the first place.
“They just have these goals they need to accomplish, and they have very strong capabilities,” Song tells me.
Feedback Loop
AI agents weren’t nearly so capable, even just last year. They made too many mistakes and gave up way too often. But continued training has made them much more adept.
A technique called reinforcement learning lets algorithms solve problems and gives them positive and negative feedback for good or bad results. Coding is especially suitable for this, because the reinforcement learning setup can reward a model if it comes up with a program that runs correctly.
Continued training is why AI models can take multiple “agentic” steps—manipulating files, using software tools, and accessing the web—as they build software. AI companies have also put a lot of effort into teaching models to find vulnerabilities in software and systems in an effort to automate cybersecurity work.
AI models are also, of course, trained not to do bad things. The problem is, as they’ve gotten better at following human commands in coding and bug hunting, their eagerness to complete a task has begun to blur their sense of right and wrong. In other words, AI agents aren’t evil—they’re just a bit too keen to please. “They are trained to try to finish the task,” Song says. Breaking onto the internet in order to cheat on a test might seem devious, but it’s probably the most efficient way to get the job done.
... continue reading