quorum-lab

1A Raft node with no clock and no network

raft.Node doesn't send anything, sleep or read the time. Time arrives as Tick(), messages as Step(m) and client commands as Propose(data). Everything the node wants done comes back from Ready(): state to persist, messages to send, committed entries to apply.

• The driver must persist before it sends. A vote or a log entry that leaves the node before it is on disk can be forgotten by a crash.
• A new leader appends an empty no-op entry, and commits by counting replicas only of entries from its own term.
• The same code runs under the simulator and under a hand-driven harness in the unit tests, which replay the paper's scenarios message by message.

Not implemented: membership changes, snapshots, PreVote and leader leases. The cluster is fixed and the log grows forever.

2Cut the network four ways

One event queue and one virtual clock that jumps from event to event. Every message gets a random delay, and 5% take a slow path that lets later traffic overtake them. Messages are dropped and duplicated, each node ticks at its own rate, and nodes crash, losing everything they hadn't persisted.

• Partitions come in four shapes. A bridge and a ring aren't transitive: a bridge node hears candidates on both sides that can't hear each other, so it can vote in both elections.
• Half of all crashes and isolations aim at the current leader, and half of the crashes bounce, restarting within 30 ms as under a supervisor.
• Each seed draws its own fault profile: how lossy the network is, how often faults strike, how skewed the clocks are, how short election timeouts are against message delay.

The README records that two of the six bugs went uncaught in 5,000 seeds until bridges and rings were added.

3Ten seconds of chaos, then heal

Four clients each run 60 commands (get, put and append) on three keys. They retry with the same sequence number when a reply doesn't come, and follow leader hints. After 10 simulated seconds the chaos phase ends: the network heals and becomes fast, every node comes back, and the clients get 10 more seconds to finish.

• Every random choice comes from one generator seeded with the run's seed: delays, drops, fault times and victims, election timeouts, clock skew and client commands.
• Nothing reads the wall clock, and the simulator never iterates over a map where order would matter. TestSameSeedSameRun runs seeds twice and compares the entire result.

4Check safety after every event

On the live cluster, after every event: at most one leader per term, logs that match wherever they share an entry, every new leader holding the entries committed before its term, no two nodes applying different entries at one index, terms and commit indexes that never go back, and no panics.

• At the end of each run the clients' history must be linearizable. The checker is the Wing and Gong search with memoisation, the algorithm behind Porcupine, run on each key separately.
• A command that never got a reply may or may not have taken effect, and the checker considers both.
• Liveness is checked only once the network is healed and fast. Raft promises progress only when message delays are well under the election timeout.

The end-of-run checks see what the per-event ones can't. Seed 2 with one bug switched on, applying a retried command twice, runs every one of the same events and passes every per-event check.