Recovery is a change of state with obligations: preserve known decisions, resolve unknown ones, and invalidate assumptions that no longer hold.
Separate restart, catch-up and disaster restore; build an evidence chain; state what remains untested.
Prerequisites: replication, unknown outcomes and ownership generations.
Restart reopens the same durable files. WAL replay and persisted metadata let the member recover its local decisions. Catch-up reconciles a lagging member with the current committed log or snapshot. Disaster restore recreates a logical cluster from a backup when the old quorum cannot return. These paths have different identities, data-loss windows and client consequences.
crash + same files → restart → catch up → read acknowledged values
lagging member → log/snapshot transfer → compare current state
quorum lost forever → backup restore → new cluster + client/epoch plan
↑ not executed hereExperiment A executes the first two paths with an intact data directory. It does not execute snapshot backup/restore, verify a recovery-point objective, or survive destruction of the host. A copied database file can omit WAL-only data; backup procedures and integrity checks belong to a separately tested restore plan.
When two voters are paused, the client times out. After recovery, the runner reads the attempted key and records whether it exists. Either outcome can be consistent with the client's earlier uncertainty. An application should use an operation identity or queryable result when the outcome matters. It should not infer “failed” from a transport deadline or replay a nonidempotent intent blindly.
Recovery also affects auxiliary state. A restored coordinator can have an older revision than a downstream fencing mark. Watch consumers may hold stale caches. Dedup records may be older than effects in an external system. Plan epochs, cache invalidation and effect/result reconciliation together; restoring just the key/value bytes is not the whole application protocol.
The evidence page renders every committed baseline repetition. results/index.json hashes the machine-readable report. Timing is descriptive and depends on local scheduling; three repetitions are mechanism checks, not a confidence interval or service-level objective. Assertions that abort a runner do not produce a new passing report.
Use the saved quorum-loss record. Write four sentences: observed client result; known pre-fault acknowledged value; post-recovery reconciliation; remaining uncertainty. Then change one assumption: the old member's disk is permanently destroyed. Explain which executed checks still apply and which require a separate restore experiment.
Observation: no success response inside the caller's deadline.
Known decision: the pre-crash acknowledged value was read after recovery.
Reconciliation: inspect unknown_write_after_recovery, not a guess.
Limit: this run did not destroy storage or partition packets.For a next experiment, add backup, deliberately remove only your disposable cluster data, restore to new owned directories, and test both contents and client epochs. The first edition provides this as a design exercise, not an executed result.
Variation is meaningful evidence of uncertainty and scheduling. Hiding it would teach the incorrect equation “timeout = rollback.”
No. It detects changes to a particular exported file. Reproduction, source inspection and a well-defined measurement method are separate requirements.
Continue with MySQL Lab for transactions and local lock evidence, Streaming Lab for end-to-end state/effect boundaries, Distributed Compute Lab for recomputation and retries, and Java Runtime Lab for interruption and caller cancellation. Those are distinct layers; none automatically supplies this chapter's distributed guarantee.