GoKawiilContext Tigris makes object storage using a database engine we built on top of FoundationDB: a distributed key-value store. JP, the founder of Ampbase, uses a control plane on Tigris with no database underneath it, and today he's going over which database behaviors he had to build himself and what that cost. Thanks JP!
Last time on the Ampbase blog I talked about all the database engines that we don't use and promised to follow up explaining what we actually do. We don't use a relational database. We use Tigris as the storage layer directly, and implement the few database behaviors we actually need on top of the two primitives it gives us. Yeah, yeah, I know; "we didn't need a database" is a catchy title that usually happens about eight (8) months before the inevitable next post being "how we tucked our tail between our legs and moved to Postgres".
In practice, when you reach for a database engine you're actually reaching for four basic features: unique constraints, transactions, indices, and history tables. In order to use Tigris' global object storage as a database, we had to implement all of these primitives ourselves. Today I'm going to peel back the curtain and show you how those primitives work so you can understand what actually goes into your database engine of choice.
Everything lives in two layers of buckets.
A global directory bucket holds the list of organizations, and every organization gets a bucket of its own. Four of those keys are doing a job a database would normally do for you, so they're labelled here and picked apart in the next section:
FIG 01 the two layers, and which primitive each key implements ┌──────────────────────────────────────────────────────┐ │ directory bucket one, global │ │ │ │ orgs/{org_id}/ │ │ metadata.json │ │ members/{sha256(email)}.json │ the index │ billing.json │ the compare-and-swap target │ channels/{channel_id}/metadata.json │ │ api-tokens/{token_id}.json │ │ events/audit/{event_ulid}.pb │ the audit log │ org-ops/queue.pb │ └──────────────────────────────────────────────────────┘ provider credentials reach this one, and only this one ┌──────────────────────────────────────────────────────┐ │ org bucket one per customer │ │ │ │ channel-slugs/{slug}.json │ the unique constraint │ channel-{channel_id}/ │ │ config-meta/{config_id}.json │ │ config-versions/{version_ulid}.json │ the history table │ bundle-meta/{bundle_id}.pb │ │ bundle-versions/{bundle_id}/{version_ulid}.pb │ │ active-config.pb │ a pointer, overwritten in place │ events/{event_ulid}.json │ └──────────────────────────────────────────────────────┘ that customer's scoped keys reach this one, and nothing else
We started out writing everything as JSON objects to each customer's bucket. After a while we started adopting more and more features to our API with protobuf options so we can define validation alongside the schema definition among other things. Marshaling and unmarshaling all the JSON got more expensive than we thought, so we switched to using Protocol Buffers directly. Our database handles both formats so if records predate the protobuf migration, everything loads as expected.
By using Protobuf, we eliminate the whole problem of managing a database layer: migrations, connections, schemas. The only downside is that Protobuf field names are forever, but to be fair it's about equally as painful to change column names in Postgres, MySQL, or SQLite.
The naive way to create a bucket per customer would be to make a bucket per customer, all in the same $bigcloud account and create a new account every time you hit a quota limit. Or have one bucket with prefixes to get around the per-account bucket limit, and rely on complexity in the IAM policy to enforce isolation. All of this sounded rather dull, and Tigris has a Partner Integration Program for exactly this shape anyway. One call to it creates a Tigris organization for that customer, its bucket, and a set of access keys scoped to it. We hold a provider identity; each customer is an organization underneath it, with strong isolation.
Isolation is baked into the infrastructure layer: no WHERE org_id = ? to forget in your app code, because the credentials that reach one customer's data cannot address anyone else's. As someone who has built a few platforms that managed databases in the past, this is the part that most people mess up.
... continue reading