GoKawiilUber has rewritten its ledger systems five times in the last ten years. And at least one of those rewrites, if not all, could have been avoided.
That’s because the root of each generation of money software at Uber was driven from bad incentives. Each started with a brand new proposal, approved as the definitive solution; in time, a fatal flaw was surfaced; and finally, a new proposal came along to replace it.
Every rewrite was someone’s promotion project.
At least one of them could’ve been avoided: the one where Uber moved to DynamoDB. In 2017, Uber launched their new payment platform on it, and the critical factor that everyone involved seemed to miss was that DynamoDB is a consumption-priced database.
You pay for every read, and every write.
With each trip generating multiple ledger entries, and Uber as a whole processing 15 million trips per day, it didn’t matter that DynamoDB was great because of high throughput at global scale. The proverbial bean counter should’ve stopped this madness from happening.
Within 2 years, the cost became prohibitely expensive:
At Uber’s scale, DynamoDB became expensive. Hence, we started keeping only 12 weeks of data (i.e., hot data) in DynamoDB and started using Uber’s blobstore, TerraBlob, for older data (i.e., cold data). TerraBlob is similar to AWS S3. For a long-term solution, we wanted to use LSG. — Migrating a Trillion Entries of Uber’s Ledger Data from DynamoDB to LedgerStore
A redesign that gets replaced 2 years later is a catastrophe.
And yet, history remembers Uber’s ledger on top of DynamoDB as a masterpiece. As late as 2024, ByteByteGo has an article praising it.
... continue reading