GoKawiilHow to ship a database every day August 14, 2026 • Tarun Pothulapati (Engineer)
Every day, turbopuffer customers ask for things: new query plans, new APIs, new index structures. In response, we deploy dozens of database upgrades per day across our clusters, many of them the same day we open the PR. Shipping fast is how we make every customer feel like they're our only customer.
We don't want to limit where you can run turbopuffer, so we support many regions across three deployment models: public SaaS, single-tenant SaaS, and BYOC. In total, we operate 100+ clusters, twice as many as we had 6 months ago, and growing as we add more public regions and many more BYOC deployments.
╔═ turbopuffer cloud account ═════════════════════╗ ╔═ customer cloud account ═══╗ ║ ║░ ║ ║░ ║ ┏━ public ━━━━━━━━━━┓ ┏━ single-tenant ━━━━━┓ ║░ ║ ┏━ BYOC ━━━━━━━━━━━━━━━┓ ║░ ║ ┃ AWS | GCP ┃ ┃ AWS | GCP ┃ ║░ ║ ┃ AWS | GCP | Azure ┃ ║░ ║ ┃ shared resources ┃ ┃ dedicated resources ┃ ║░ ║ ┃ customer's resources ┃ ║░ ║ ┃ ┃ ┃ ┃ ║░ ║ ┃ ┃ ║░ ║ ┃ tpuf operator ┃ ┃ tpuf operator ┃ ║░ ║ ┃ no tpuf operator ┃ ║░ ║ ┃ access ┃ ┃ access ┃ ║░ ║ ┃ access ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━┛ ┗━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ║ ║░ ║ ║░ ╚═════════════════════════════════════════════════╝░ ╚════════════════════════════╝░ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ╔═ tpuf account ═══════════╗ ║ ┏ public ━━━━━━━━━━━━━━┓ ║ ║ ┃ AWS | GCP ┃ ║ ║ ┃ shared resources ┃ ║ ║ ┃ tpuf operator access ┃ ║ ║ ┗━━━━━━━━━━━━━━━━━━━━━━┛ ║ ║ ┏ single-tenant ━━━━━━━┓ ║ ║ ┃ AWS | GCP ┃ ║ ║ ┃ dedicated resources ┃ ║ ║ ┃ tpuf operator access ┃ ║ ║ ┗━━━━━━━━━━━━━━━━━━━━━━┛ ║ ╚══════════════════════════╝ ╔═ customer account ═══════╗ ║ ┏ BYOC ━━━━━━━━━━━━━━━━┓ ║░ ║ ┃ AWS|GCP|Azure ┃ ║░ ║ ┃ customer's resources ┃ ║░ ║ ┃ no tpuf operator ┃ ║░ ║ ┃ access ┃ ║░ ║ ┗━━━━━━━━━━━━━━━━━━━━━━┛ ║░ ╚══════════════════════════╝░ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░
The problem with BYOC
BYOC clusters live inside our customers' cloud accounts, to which we hold no credentials by default. We can't just SSH or kubectl in. Some BYOC vendors solve this by asking the customer to carve out a dedicated cloud account and grant the vendor standing admin credentials inside it. That keeps the rest of the customer's cloud account isolated, but every dedicated account creates security monitoring + compliance + billing overhead that we generally prefer to avoid.
We don't want different control planes for BYOC and SaaS, so we have to design for the lowest common denominator. We must be able to operate every cluster without reaching in.
How do you operate a database cluster you can't touch?
The only way this works is if every operation we need to perform on a cluster can run without us reaching in. The cluster must be able to independently drive its operations to a terminal state, even if it loses its connection to the central control plane.
The solution to this is standard Kubernetes stuff. On every cluster, we run a local cluster agent that implements a simple state machine. A single Kubernetes CRD called TurbopufferOperation expresses every operation kind, from upgrade to tidy . A Kubernetes controller drives each custom resource (CR) from state to state via a reconciliation loop until it reaches a terminal state. Operations advance on their own by default, but BYOC customers can gate operations on approval or maintenance windows. These are baked into the CRD as waiting states that advance when approvals are given or the window opens.
... continue reading