This year marks a decade for me in observability.
I left my engineering job in 2016 to start Timber.io, a hosted logging platform, because I thought logs could be simple and great. Timber became Vector. Vector got mass adoption. It got acquired, and I stayed for three years.
And somewhere along the way, the optimism curdled.
I'm not a cynical person. I believed observability could make engineers' lives better. But after a decade, after hundreds of conversations with teams bleeding money across every major vendor, after hearing firsthand how their vendors strong-armed them instead of helping; I've seen enough. The whole industry has lost the plot.
Does any of this sound familiar?
You run observability at your company. But really, you're the cost police. You wake up to a log line in a hot path, a metric tag that exploded cardinality. You chase down the engineer. They didn't do anything wrong, they're just disconnected from what any of this costs. The renewal is always in the back of your mind because mismanaging it reflects poorly on you. Sometimes you catch these mistakes. Sometimes you don't. When you don't, you crawl to your rep asking for forgiveness. Maybe they help the first time, even the second. By the fourth or fifth, they stop. "It's your data." But even with the mistakes, if you're diligent, checking dashboards, staying on top of things, you manage to stay under your commit and avoid an early renewal. But the renewal still gives you a black eye: 40% higher than last year. Your budget didn't grow that much. So you consider switching vendors, but asking your engineers to frantically migrate dashboards, alerts, and change workflows is a distraction that also reflects poorly on you. You're in a lose-lose situation. So you go back to your vendor and ask them to help. You championed them internally; brought them six, seven figure business. Surely they'd return the favor. A slightly bigger discount, help you cut costs by showing you what data is safe to drop. But they don't budge. They could help; they don't.
Case Taintor, Director of Engineering at Klarna, put it all too well:
The most frustrating part of watching your money burn is knowing your supplier could help if they only cared about your long term success.
So why has this gone on for over a decade? Something is deeply wrong if after ten years these same problems not only exist, but have gotten worse.
But what's wrong, exactly? Should your vendor help you? It is your data. They didn't create it. You sent it to them under their pricing model. For years I accepted that framing too. Maybe this is just how it works.
... continue reading