Consistency Over Availability: How rqlite Handles the CAP Theorem
rqlite is a lightweight, user-friendly, open-source, distributed relational database. It’s written in Go and uses SQLite as its storage engine.
When it comes to distributed systems the CAP theorem is an essential concept. It states that it’s impossible for a distributed database to simultaneously provide Consistency, Availability, and Partition tolerance. The challenge is in the face of a network partition, a database can only be available or consistent, but not both.
Let’s take a look at the CAP theorem and see how rqlite fits into this fundamental trade-off.
Understanding Consistency, Availability, and Partition Tolerance
Consistency (C): A consistent system ensures that all nodes in a distributed cluster have the same data at the same time. This means any read request will return the most recently written data.
A consistent system ensures that all nodes in a distributed cluster have the same data at the same time. This means any read request will return the most recently written data. Availability (A): An available system ensures that every request receives a response, regardless of network issues. The system continues to operate even if some nodes are unreachable, though there’s no guarantee the data will be the most recent.
An available system ensures that every request receives a response, regardless of network issues. The system continues to operate even if some nodes are unreachable, though there’s no guarantee the data will be the most recent. Partition tolerance (P): A partition-tolerant system continues to function even when network failures partition the system into isolated groups of nodes.
When a network partition occurs, a choice must be made. Do you prioritize consistency by not responding to requests on the “bad” side of the partition, or do you prioritize availability by continuing to serve requests, even if it means potentially serving stale data? This is the core trade-off between CP and AP.
CP vs. AP Systems
A CP (Consistency-Partition tolerant) system prioritizes consistency. If a network partition occurs, the system will block writes on the minority side of the cluster to prevent data inconsistencies. This ensures that any data written is consistent, and when the partition is resolved, the system synchronizes without conflicts.
An AP (Availability-Partition tolerant) system prioritizes availability. It continues to accept write requests on both sides of a partition, even though it risks data divergence. After the partition is healed, the system must deal with potentially conflicting data.
Where rqlite Fits: A CP System
rqlite is a CP system. It’s built on the Raft consensus protocol, which inherently prioritizes consistency. If a network partition occurs, an rqlite cluster will remain available only on the side of the partition that contains a majority of nodes. The other side will stop accepting write requests.
This design guarantees that any data written to the majority side remains consistent across all available nodes. Once the network partition is resolved, the minority nodes are updated, and the cluster resumes normal, consistent operation. This CP approach ensures data correctness and integrity, making rqlite an important tool for applications where data consistency is a core requirement.
Read Consistency: Beyond the CAP Theorem
While rqlite is fundamentally a CP system, it gives you fine-grained control over the C and A trade-off through its selectable read consistency levels: weak, linearizable, strong, and none. These levels allow you to balance performance with data correctness, and choose the best trade-off for your application. Best of all, you can even choose your consistency level on a per-read request basis.
Weak: Great but not Perfect
Weak consistency is rqlite’s default and is typically the right choice for most applications. With weak consistency a node receiving a read request checks if it’s the leader and, if so, reads its local SQLite database. And if the node is not the leader it will transparently forward the request to the leader, waiting for the response before returning to the client. The end result is very fast reads, almost always perfectly consistent.
However, with weak, there is a very small window of time where a node may think it’s the leader after it’s been deposed, and a write has taken place, potentially resulting in a stale read. While this risk is minimal in a stable cluster, it’s an important consideration. Interestingly this was one of the very first issues fixed in rqlite.
Strong: A Testing Tool, Not for Production
Strong consistency provides the highest level of data freshness by sending the read request through the Raft log itself. This ensures all committed entries are applied before the read is executed. While this removes any doubt about data staleness, it comes with a significant performance penalty, making it generally unsuitable for production use. It is primarily useful for certain testing scenarios, and its use is a key pattern in the rqlite test suite.
Linearizable: The Best of Both Worlds
For applications where up-to-date data is critical but the performance cost of strong consistency is too high, linearizable reads are the answer. This method ensures that the data returned is absolutely current by having the leader first confirm its status with a quorum of followers. This provides the same strong consistency guarantees as strong reads but with significantly less latency, making it the ideal solution for both performance and data correctness.
None: Max Speed, (Almost) Zero Guarantees
With none consistency, the node receiving the read request simply queries its local SQLite database without any Leadership or cluster-connection checks. This is the fastest possible read, but there are no guarantees about the data’s freshness. The node could be completely disconnected from the rest of the cluster and still serve the request. This level is particularly effective when used with read-only nodes, where you can combine it with freshness parameters to add a layer of safety, ensuring a node hasn’t been out of contact with the leader for too long.
Next Steps