GoKawiilThe third time
I’ve built the same system three times now, at three different companies, for three different providers. It never has a name and it never appears on a roadmap, but it always goes the same way: the truth about your own customers lives in someone else’s database. The users live in an identity provider, the subscriptions in Stripe, the bounces in whatever sends your email, and your product needs that truth locally. So you subscribe to webhooks and keep a copy.
The first time, I thought I was building an endpoint: one route that parses the JSON and updates a row, an afternoon of work.
The afternoon grew into a week. First came signature verification, because an open endpoint that mutates your database is a hole. Then the dedup table, because deliveries arrive twice and the docs cheerfully call this “at-least-once.” Then the handler got a buffer, because a membership.created sometimes shows up before the user.created it points to. Then the bootstrap importer, because webhooks only tell you what happens after you subscribe, and it raced the live events, so it grew a locking scheme. And finally came the reconciliation cron: a job that crawls the provider’s list APIs at 3 a.m., diffs them against our tables, and quietly fixes what disagrees.
I want to be honest about what that cron is. It’s a written confession. It says: I do not trust the copy I built, and I have no way to know when it’s wrong, so I will re-derive it from scratch every night, forever.
The trust was gone for a reason. The drift never announces itself; ours was found by a support ticket. A customer had cancelled months earlier and our database still said active ; some customer.subscription.deleted had evaporated between Stripe and us, and nothing anywhere was capable of noticing: not their dashboard, which showed the delivery as retried and eventually dropped, and not our logs, which cannot log a request that never arrived.
And that’s only the code. Every provider also brings its own dashboard. Three providers, three webhook configuration pages, each with its own idea of how endpoints are registered, which events exist, how test and live environments are kept apart, and where the signing secret lives. When something breaks, debugging is a tour: their delivery log in one tab, our logs in another, and a third tab for whichever dashboard I currently suspect. None of them look alike, and all of them have to be checked.
By the third time I built this system, I had stopped pretending. I budgeted for the whole stack up front (signatures, dedup, buffering, bootstrap, cron) and somewhere in the middle of writing my third dedup table, I finally asked the question I should have asked the first time.
What exactly am I reconstructing here?
Notifications aren’t data
... continue reading