In Paxos, a value is confirmed only if it is accepted with the same ballot by a majority of acceptors. Why is having a majority of acceptors accept the same value not enough?

(At first, I thought the accept action in Paxos corresponds to decide in Ben-Or’s algorithm (Ben-Or, 1983), and thus I was wondering why Ben-Or’s algorithm can let nodes decide a value in different rounds, but Paxos requires a majority of nodes to accept with the same ballot. However, this comparison is wrong. Ben-Or’s decide is more like the behavior of learners in Paxos, and Ben-Or’s algorithm can be regarded as skipping the first phase of Paxos if we have to parallelize them.)

To answer this question, we can simply construct a counterexample which shows that, after a majority of acceptors have accepted a same value \(v\) with different ballots, the system may decide a different value \(v'\). I will use \((n, v)\) to denote a proposal of value \(v\) with ballot \(n\).

Let’s use the simplest setting with 3 acceptors \(A_1, A_2, A_3\), where \(A_1\) has accepted \((n_1, v)\) and \(A_2\) has accepted \((n_2, v)\). Without losing generality, let \(n_1 < n_2\). Now we want one of them to switch to accept another value \((n', v')\) (after which another proposer can let the system accept \(v'\) by forming a majority with \(A_3\)). For simplicity, we assume there are no other concurrent proposals.

If the prepare message of $n’$ arrives after \(A_1\) and \(A_2\) accept \(v\), its proposer will learn and propose \(v\), so at least one of \(A_1\) and \(A_2\) must prepare \(n'\) before accepting \(v\). On the other hand, \(n'\) cannot be smaller than both \(n_1\) and \(n_2\), otherwise they will ignore the accept message for \(v'\), so we have \(n_1 < n' < n_2\).

We visualize the construction below. A new proposer can use \(A_1, A_3\) to form a majority, and let the system accept \(v_2\) at the end, even if two acceptors have accepted \(v_1\) with different ballots at some time.

Overall, in Paxos, although an acceptor will try to let subsequent proposers follow its accepted value, the proposer can find a majority/quorum without including this acceptor. Even if a majority of acceptors accept the same value \(v\), if they are not of the same ballot, another proposer can still succeed in letting a majority of acceptors prepare a ballot that lies in the gap among the accepted ballots for \(v\) before they all accept \(v\).

This makes me think about another possible design. What if we can have a method that makes there no gap between the ballots for accepting the same value? I did not conduct a formal proof, but it seems to be enough for correctness. However, how can we ensure there are no new ballots can be inserted? Using consecutive integers? Using
timestamps and safe timestamp mechanism?

Another thought is about the proposed value. In Ben-Or’s algorithm, a node tries to follow the value from other nodes in its Phase-1 in order to facilitate convergence. In contrast, in Phase-1 of Paxos, the proposer does not follow the value prepared by acceptors; its proposed value is affected only by accepted values.

What if we adapt Paxos and let a proposer follow the value observed in Phase-1. I was thinking this may make it possible to enable Paxos to confirm a value by accepting it value with different ballots. However, by simple variants of the previous construction, we can see that it is not correct. However, will this adaption cannot make the consecutive-ballot design viable?

Anyway, these are just some random thought, and I did not spend much time to verify them, as the vanilla Paxos (Paxos-made-simple without a stable leader) is rarely used in deployments.

References:

1. Leslie Lamport . Paxos made simple. ACM SIGACT News (Distributed Computing Column) 32, 4 (Whole Number 121, December 2001) 2001
2. Miguel Castro,  Barbara Liskov,  and others . Practical byzantine fault tolerance. In OsDI 1999
3. Michael Ben-Or . Another advantage of free choice (extended abstract) completely asynchronous agreement protocols. In Proceedings of the second annual ACM symposium on Principles of distributed computing 1983

Spanner in a Nutshell

Spanner is a geo-distributed database built by Google. Spanner has a data model that is stronger than key-value stores but weaker than fully relational databases.
Spanner adopts a layered design, with concurrency control (2PL) on top of replication (Paxos). This design is easy to maintain, but it may cause extra RTTs. Spanner has three types of transactions, namely read-write (RW) transactions, consistent read-only transactions (RO), and stale snapshot transactions (the namings may be a little bit different from its original paper).

The key weapon of Spanner is its TrueTime API based on atomic clocks and GPS. The TrueTime API has a (at least practically) guaranteed offset among clocks. Compared to traditional non-synchronized clocks, a node in Spanner can predicate that a remote node’s clock has passed \(T\) without communication, which brings Spanner unique strength on efficiently ensuring consistency.

Spanner’s Consistency/Isolation

For read-write (RW) transactions, Spanner uses (strict) 2PL for concurrency control and uses leader-based Paxos for replication. A RW transaction is triggered by its client: the client first acquires all read locks, reads all values, and buffers all writes; then, a client acquires all write locks and commits via 2PC. This is a standard 2PL, and it actually already ensures Strict Serializability (even with replication, as all read/write requests go through the leader) without TrueTime. A key invariant when assigning timestamp is that the timestamp order should be the same as the equivalent serial order of 2PL.

For read-only (RO) transactions, Spanner uses TrueTime to ensure their consistency. For consistency, there are two main requirements for a RO transaction \(R\) at timestamp \(ts\). First, Spanner needs to ensure that \(R\) sees all writes with timestamps smaller than \(ts\). This completeness enures atomicity because updates from the same transaction share the same timestamp, so completeness already infers atomicity. The completeness also preserves serializability: RW transactions are serialized according to their timestamps, so completeness ensures that there is no gap in the equivalent serial schedule of RW transactions. Completeness together with the second requirement below also ensures the strictness of serializability.

Second, to ensure “strictness” in strict serializability, Spanner must ensure that a RO transaction’s timestamp is larger than all completed RW transactions’. This is where Spanner’s TrueTime comes to help. Since nodes’ clocks have a bounded offset, consider a RO transaction \(R\) starting at \(t\). When \(R\) starts, any other node’s clock is at most \(t +\frac{1}{2} offset\). Therefore, assigning \(R\) with a timestamp larger than \(t +\frac{1}{2} offset\) is enough for consistency.

Without TrueTime, these two requirements become expensive to meet. For the first requirement, Two key aspects are (1) to ensure that nodes will not travel back in the history and assign transactions with past timestamps, and (2) to have a mechanism helping nodes to predicate that all on-the-flight transactions. Achieving these two aspects efficiently in a geo-distributed deployment is challenging. Ocean Vista (Fan & Golab, 2019) is an impressive work showing how these two aspects can be met with commodity hardware while still ensuring high performance. My previous work DAST (Chen et al., 2021) also addresses these two aspects, but in a more specific near-client computing deployment scenarios: data is assigned home regions based on geo-locality, which brings DAST the potential for higher performance.

For the second requirement, without TrueTime, a system must keep track of the timestamp of finished transactions. This cannot be efficiently achieved in general cases for geo-distributed deployments, and normally a RO has to wait/communicate with delay in the order of global RTT time. This is actually in line with the SNOW (Lu et al., 2016) impossibility result, and explains why Google invest to build such a strong hardware to support RO transactions. There are two difficulties here. First, nodes’ clocks are not synchronized, so reading at a specific timestamp is bounded by the global straggler (who can assign relevant read-write transactions). DAST somehow sidesteps this difficulty because in DAST, for each key \(K\), only a restricted set of local nodes can assign timestamps to RW transactions accessing \(K\). Second, most systems do not explicitly record the time when each transaction finishes, so to ensure strict serializability, the system has to include a conservative superset of finished transactions (e.g., those who are assigned timestamps). This superset makes the RO must wait for un-finished transactions.

Overall, there seems to be two chances for improving RO transactions in Strictly serializable databases: (1) strong data ownership based on locality (e.g,. SLOG (Fan & Golab, 2019)), and (2) leveraging strongly synchronized clocks (e.g., recent works on NSDI (Geng et al., 2018; Najafi & Wei, 2022) )

References:

1. Ali Najafi,  and Michael Wei . Graham: Synchronizing Clocks by Leveraging Local Clock Properties. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22) 2022
2. Xusheng Chen , Haoze Song , Jianyu Jiang , Chaoyi Ruan,  Cheng Li , Sen Wang,  Gong Zhang,  Reynold Cheng , and Heming Cui . Achieving low tail-latency and high scalability for serializable transactions in edge computing. In Proceedings of the Sixteenth European Conference on Computer Systems 2021
3. Hua Fan,  and Wojciech Golab . Ocean vista: gossip-based visibility control for speedy geo-distributed transactions. Proceedings of the VLDB Endowment 2019
4. Yilong Geng,  Shiyu Liu,  Zi Yin,  Ashish Naik,  Balaji Prabhakar,  Mendel Rosenblum,  and Amin Vahdat . Exploiting a natural network effect for scalable, fine-grained clock synchronization. In 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18) 2018
5. Haonan Lu,  Christopher Hodsdon,  Khiem Ngo,  Shuai Mu,  and Wyatt Lloyd . The SNOW Theorem and Latency-Optimal Read-Only Transactions. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16) 2016
6. James C Corbett,  Jeffrey Dean,  Michael Epstein,  Andrew Fikes,  Christopher Frost,  Jeffrey John Furman,  Sanjay Ghemawat,  Andrey Gubarev,  Christopher Heiser,  Peter Hochschild,  and others . Spanner: Google’s globally distributed database. ACM Transactions on Computer Systems (TOCS) 2013

The definition of 2PL is simple and clear. A transaction in 2PL is not able to grab locks once it starts to release locks. Consequently, the transaction’s life time can be divide into an expanding phase where the transaction keeps grabbing new locks, and a shrinking phase where the transaction keeps releasing locks. Intuitively, between these two phases, there will be a period of time where the transaction is holding all its locks and conducting its operations.

Let’s use \(H(T)\) to represent the time period of transaction $$T$ holding all its locks (we do not need to distinguish the start/end event of this period).

Consider two conflicting transactions \(T_1\) and \(T_2\). 2PL ensures that \(H(T_1)\) and \(H(T_2)\) has no overlap; because otherwise, during the overlapped time, \(T_1\) and \(T_2\) both hold the locks of their shared keys, breaking the semantic of locks.

Say \(H(T_1)\) is before \(H(T_2)\) in real time. Then, \(T_2\) must grab the locks of shared keys after \(T_1\) releasing them, i.e., \(T_2\) is serialized after \(T_1\), denoted by \(T_1 <_S T_2\). The direct proposition here is that 2PL ensures serializability: if there is a cycle in the dependency graph \(T_1 <_S .... <_S T_1\), it means that \(T_1\) grab locks after \(T_1\) release them, violating 2PL’s definition.

2PL also ensures strict serializability, which can be proved by contradiction. Suppose \(T_x\) finishes before \(T_y\) starts, but the serial order is \(T_y <_S ....<_S T_x\). This means \(T_x\) grabs some locks after \(T_y\) releasing them, contradicting with the assumption.

Therefore, 2PL is already enough for Strict Serializability.

TODO: S2PL

For single key, no-stale-reads considers each read independently, while linearizability also restricts each read value w.r.t. other reads. For multiple keys, things become a little bit complicated the overall theme is the same.

Background

When doing my projects, many times I got confused with the guarantee of “the system provides no stale reads” (i.e., regularity) and “the system preserves all real-time ordering requirements” (i.e., linearizability). I found two constructions to distinguish them. The first construction (from (Overflow, n.d.) is for single-key operations as shown below, where wwww and rrr means the duration of one single write/read operation.

Key(x);  0                  1
P1:         wwwwwwwww(x=1)
P2:                rr(0)        
P3:         rr

In this construction, P3’s read may return 0 or 1 if the system ensures “no stale reads” because the read is concurrent to the write. However, P3’s read must return 0 as it precedes P2’s read in real-time: P3’s read must always be ordered before P2’s read in any history/schedule. Symmetrically, if P3’s read returns 1, then P2’s read must not return 0.

The second construction is for transactional systems (the construction in my DAST paper (Chen et al., 2021)). Consider the following construction with two keys A and B. T2 and T3 only access one key each, while T1 is a distributed transaction accessing both keys. In this construction, we have $T2 -> T3$ in real-time. However, the serializable order (i.e., the equivalent serial schedule for serializability) among these three transactions is $T3->T1->T2$, which breaks the real time-order. This construction does ensure no stale reads, but does not preserve all real-time ordering requirements.

A:   T1    T2
B:              T3    T1

Recently, I read the RSS paper (Helt et al., 2021) , which helps to shed light on this problem. RSS ensures all reads see finished writes (i.e., no stale reads) and preserves the real-time ordering from write operations/transactions to other operations. RSS is proved to be invariant-equivalent to linearizability. To understand the idea of invariant-equivalence, we need to first distinguish the definitions of invariants and anomalies. Usually, invariants are defined from system/service’s pov, while anomalies are defined from the external view. For instance, a system/service cannot track whether two processes/clients contacts externally (e.g., Alice calls Bob after its operation finishes). Therefore, we usually refer to strict serializability and linearizability as external consistency: the system/service is consistent even if one views the external world altogether where processes/clients can communicate with no delay.

RSS paper writes, “like all prior regular models, (RSS ensures that) reads must return a value at least as recent as the most recently complete conflicting writes”. This sentence motivates me to dig deeper into the classical papers about consistency levels.

Shared Registers

The basic single-write multiple-writer shared register model was first proposed by Lamport in 1986 (Lamport, 1986), together with the atomic, regular, and safe consistency models. This model only has one writer and one key, which serve as the base for all subsequent consistency models.

• A safe register ensures that a read not overlapping with writes returns the latest writes; otherwise, the reads can return any value (even a corrupted one). Note that in the basic model, there is only one writer, so no concurrent writes. Such registers may show in shared registers on chips (Lamport, 1986).

• A regular register is a little stronger. It also requires that reads must return the value that is at least as recent as latest finished writes, while it can freely choose whether to return the values of concurrent writes.

• An atomic register provides the strongest guarantee. In addition to previous requirements, it also requires that a subsequent read cannot returns a earlier value than a read finished before it (i.e., linearizability).

Back to the motivation cases.

It is obvious that in the first construction, a consistency model ensuring this can be called as regular, like in RSS. Note that this is not Monotonic Read guarantee, as MR guarantee is defined only within session. It is also sequential, but not linearizability.

In the second construction, however, things are a little complicated as the original model only involves one register. In essence, the difference between the two constructions is that P3’s read is read-only, while T2 is not.

The key ambiguity sources from the fact that Lamport’s original paper only defines regularity for single register: read must see a version at least as recent as the latest finished writes. However, this construction has two registers, but the definition does not discuss the relation of two (unrelated) writes on two different registers. As far as I am concerned, there is only one work (Viotti & Vukolić, 2018) that defines regularity for multiple keys with the following three requirements:

• There is an equivalent total order (a.k.a., the arbitration order) of all requests.
• This order respects the real-time order (a.k.a., the “return before” relation) from writes to any operations.
• The system is return value consistent, i.e., the return value is as expected. This is a very weak guarantee, but the atomic register model does not ensure this.

This definition also requires the real-time ordering from a write to an un-related operation must also be preserved. The RSS paper follows this definition. If we adopt this definition, the second construction is not regular, as it does not the real-time ordering between T2 and T3 if T2 contains write operations.

However, this general definition has not reached an agreement in academia/industry, as the intuition behind Lamport’s definition of “regularity” is that the register has no stale reads: all finished writes must be visible to subsequent reads until it is overwritten.

I will discuss more regular consistency models defined in (Shao et al., 2011) on multi-writer regularity in subsequent posts.

References:

1. Jeffrey Helt,  Matthew Burke,  Amit Levy,  and Wyatt Lloyd . Regular Sequential Serializability and Regular Sequential Consistency. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles 2021
2. Xusheng Chen , Haoze Song , Jianyu Jiang , Chaoyi Ruan,  Cheng Li , Sen Wang,  Gong Zhang,  Reynold Cheng , and Heming Cui . Achieving low tail-latency and high scalability for serializable transactions in edge computing. In Proceedings of the Sixteenth European Conference on Computer Systems 2021
3. Paolo Viotti,  and Marko Vukolić . Consistency in non-transactional distributed storage systems. ACM Computing Surveys (CSUR) 2018
4. Cheng Shao,  Jennifer L Welch,  Evelyn Pierce,  and Hyunyoung Lee . Multiwriter consistency conditions for shared memory registers. SIAM Journal on Computing 2011
5. Leslie Lamport . On interprocess communication. Distributed computing 1986
6. Stack Overflow . https://stackoverflow.com/questions/8871633/whats-the-difference-between-safe-regular-and-atomic-registers. 1986