Skip to content
#Software

Consistency Tradeoffs in Modern Distributed Database System Design (Or whitepaper about PACELC.)

#Software #Architecture #DistributedSystems #SystemDesign

Continuing a series of posts about consistency (CAP theorem, proof)I'll tell you about whitepaper 2012 Daniel J. Abadi extends the CAP theorem to PACELC (reads[pass-elk]). The essence of this article sounds like this.

The CAP theorem’s impact on modern distributed database system design is more limited than is often perceived. Another tradeoff—between consistency and latency—has had a more direct influence on several well-known DDBSs. A proposed new formulation, PACELC, unifies this tradeoff with CAP.

The CAP theorem focuses on the behavior of a distributed system during network partitioning. But most of the time, the system works in a state where everything is fine with the network. So it’s interesting to see what kind of tradeoff this is. And this is the triangle "consistency, availability, and latency", which the author simplifies to a pair of consistency and latency.

Availability and latency are arguably the same thing: an unavailable system essentially provides extremely high latency The author further highlights the need replication from the requirement of high availability and probability of failure in the distributed system. It does this from the opposite, that is, if you do not do this, then with a sufficiently long system life, one of the components of the system will fail, its data will be lost, and this will effectively lead to their inaccessibility.

Next, the author considers the options replication

  1. Data updates sent to all replicas at the same time
  2. Data updates sent to an agreed-upon location first
  3. Data updates sent to an arbitrary location first

  1. Here, sending updates occurs on all nodes. There are two options. There is no pre-processing layer - consistency will be broken here. (linearizability) by Gilbert and Lynch (already mentioned proof of the CAP theorem). -b. There is such a layer - it will take time to coordinate nodes if there are many, and if there is one node in preprocessing, then we will spend time sending requests to it, which in a geo-distributed system can be costly.

  2. Here we have a master node. (For different data elements, we may have different masters.). And this master node flies all the update requests that it performs and effectively determines the order of operations, which is the same for all the replicas. It's here. 3 replicability

  1. Synchronous replication When we update, we wait until all the lines are updated. It is consistent but increases latency.
  2. Asynchronous replication - here we usually store update in persistent storage, but we do not wait for successful replication and confirm the operation. It is interesting to see how we read the data in this case.
  • a. If we read from the master, then we have no problem with consistency, but latency can be big, since the master is far away. Plus, with a high load on the master, we also get problems with latency. -b. If we read from replicas, we can potentially read outdated data.
  1. Synchronous and asynchronous combination In fact, this is something from the series of tunable consistency in Cassandra, where you can specify a quorum for writing and reading.
  1. This is similar to the second point, but we send updates not to a specific wizard for a particular element, but to an arbitrary node. Problems with consistency and latency are similar to those in the second paragraph.

Further, the author analyzes how these tradeoffs work in the popular databases at the time: Dynamo, Cassandra, PNUTS, Riak, ... and it is concluded that the design of solutions in databases often focuses more on consistency/latency under normal conditions, rather than consistency/availability under the onset of partition. And then the author proposes to rewrite CAP to PACELC with approximately this wording.

If there is a partition (P), how does the system trade off availability and consistency (A and C); else (E), when the system is running normally in the absence of partitions, how does the system trade off latency (L) and consistency (C)?

#Software #Architecture #DistributedSystems #SystemDesign