How I Dropped the Production Database on a Friday Night The case for moving fast and breaking things (before your competitors kill you)
The worst 7PM in my software engineering career
Picture this: It's a Friday evening in Paris, and I'm wrapping up what should have been a routine week at Joe AI, the real-estate startup where we were building an AI agent that automated communications for property developers across France.
I had just finished what I thought was a clean migration: moving our entire database from our old setup to PostgreSQL with Supabase. I had carefully designed a normalized schema with all the proper foreign keys to enable clean join queries. We were processing thousands of leads for a major French real-estate developer, and everything seemed to be humming along nicely.
Now, here's where I should mention a small detail: we were developing directly on the production database. You heard that right. No staging environment, no safety nets. Just pure, unfiltered startup velocity. We were early-stage, moving fast, and honestly? It felt like the right trade-off at the time.
That Friday, our big client couldn't update their user profile. Simple fix, I thought. I'll just delete their user record and recreate it with the correct fields. Clean slate, fresh start.
What I had forgotten, and this is where my "careful" database design came back to haunt me, was that our entire architecture relied on Row Level Security (RLS). Think of RLS as a security guard for your database: it controls which rows each user can access based on rules you set up. For example, you might create a rule that says 'users can only see leads assigned to their organization'. In our case, every single piece of data—leads, conversations, analytics, everything—was linked back to the users table through foreign keys to avoid unwanted access from other unauthorized users. This wasn't just for organization; it was our security model.
Example of a relational database schema where most entities are linked to the users table to restrict access to rows in the database. This example is taken from my simple auction SaaS Rankbid.
I clicked delete on that user record.
The database started acting strange immediately. Error messages from Supabase began flooding in in the top right corner of my screen. Surprised, I refreshed the page, hoping it was just a temporary glitch.
... continue reading