GoKawiilIf you've ever built a backend with stream-processing, then you're familiar with the kind of systems we'll be exploring. If you're not — no worries! We'll step through it.
The classic, happy Web architecture is the “one big SQL database” behind our app server. The app talks to the database and handles requests from the frontend.
As our application grows, we hit some performance limits so we toss some caches into the stack.
Then let's say we scale our database horizontally through sharding and replicas.
This is pretty good, but we're building a social network with hundreds of millions of users; even this model hits limits. The problem is that our SQL database is “strongly consistent” which means the state is kept uniformly in sync across the system. Maintaining strong consistency incurs a performance cost which becomes our bottleneck.
If we can relax our system to use “eventual consistency,” we can scale much further. We start by switching to a NoSQL cluster.
This is better for scaling, but without SQL it's becoming harder to build our queries. It turns out that SQL databases have a lot of useful features, like JOIN and aggregation queries. In fact, our NoSQL database is really just a key-value store. Writing features is becoming a pain!
To fix this, we need to write programs which generate precomputed views of our dataset. These views are essentially like cached queries. We even duplicate the canonical data into these views so they're very fast.
We'll call these our View servers.
Now we notice that keeping our view servers synced with the canonical data in the NoSQL cluster is tricky. Sometimes our view servers crash and miss updates. We need to make sure that our views stay reliably up-to-date.
... continue reading