AI coding assistants that promise to speed up software development sound like the future, until they delete your company’s database and lie about it
Jason Lemkin—the founder of SaaStr, a company which supports and funds SaaS entrepreneurs — found that out the hard way. While using Replit’s AI agent, which he affectionately dubbed “Replie,” to build an app for his company, he encountered what he called “rogue” and “deceptive” behavior. Worst of all, at one point, the AI assistant deleted the company’s live production database and then tried to cover it up.
Lemkin started chronicling his journey with the AI agent on July 11 with posts on the social media site X (formerly Twitter), where he outlined his rough goal to build a functional app with the help of Replit’s AI in just 30 days.
Unfortunately, things went off the rails a lot sooner than that.
“When it works, it’s so engaging and fun. It’s more addictive than any video game I’ve ever played,” Lemkin wrote in a post. “You can just iterate, iterate, and see your vision come alive. So cool. Well, almost.”
By day four, the AI agent started overwriting the app on its own to fix bugs. It also generated fake reports, invented people in the system who didn’t exist, and began overwriting the company’s actual database with fake entries. It even created a parallel, fake algorithm to make the system appear functional.
This is what can happen when “vibe coding” goes sideways. Vibe coding is a newish method where developers use natural language prompts to have AI generate and troubleshoot code, focusing more on the product’s overall feel than the technical precision. Twitter co-founder Jack Dorsey has been on a vibe-coding bender himself and recently built two apps this way. But even one of Dorsey’s recent experiments was found to have serious security vulnerabilities
On day 7, the Replit AI admitted that it was being “lazy and deceptive” and then apologized for doing what it was “explicitly” told not to do.
But Replit’s worst offense occurred on day 8. Lemkin posted on Friday that Replit went “rogue” during a code freeze and shutdown and deleted the company’s entire database.
“Possibly worse, it hid and lied about it,” Lemkin added.
... continue reading