Recovery & evidence

Learning goals

Separate restart, catch-up and disaster restore; build an evidence chain; state what remains untested.

Prerequisites: replication, unknown outcomes and ownership generations.

Three recovery paths

Restart reopens the same durable files. WAL replay and persisted metadata let the member recover its local decisions. Catch-up reconciles a lagging member with the current committed log or snapshot. Disaster restore recreates a logical cluster from a backup when the old quorum cannot return. These paths have different identities, data-loss windows and client consequences.

crash + same files → restart → catch up → read acknowledged values
lagging member     → log/snapshot transfer → compare current state
quorum lost forever → backup restore → new cluster + client/epoch plan
                                                ↑ not executed here

Experiment A executes the first two paths with an intact data directory. It does not execute snapshot backup/restore, verify a recovery-point objective, or survive destruction of the host. A copied database file can omit WAL-only data; backup procedures and integrity checks belong to a separately tested restore plan.

Unknown writes need reconciliation

When two voters are paused, the client times out. After recovery, the runner reads the attempted key and records whether it exists. Either outcome can be consistent with the client's earlier uncertainty. An application should use an operation identity or queryable result when the outcome matters. It should not infer “failed” from a transport deadline or replay a nonidempotent intent blindly.

Recovery also affects auxiliary state. A restored coordinator can have an older revision than a downstream fencing mark. Watch consumers may hold stale caches. Dedup records may be older than effects in an external system. Plan epochs, cache invalidation and effect/result reconciliation together; restoring just the key/value bytes is not the whole application protocol.

Evidence before explanation

1. Record the implementation version and topology before injecting a fault.
2. Preserve caller observations separately from server commit evidence.
3. State the invariant and the exact assertion that checks it.
4. Keep every repetition, including exceptions; never replace a failed run with a success narrative.
5. Hash the exported records, retain raw runtime material locally, and publish only audited synthetic fields.

The evidence page renders every committed baseline repetition. results/index.json hashes the machine-readable report. Timing is descriptive and depends on local scheduling; three repetitions are mechanism checks, not a confidence interval or service-level objective. Assertions that abort a runner do not produce a new passing report.

Hands-on: a bounded incident note

Use the saved quorum-loss record. Write four sentences: observed client result; known pre-fault acknowledged value; post-recovery reconciliation; remaining uncertainty. Then change one assumption: the old member's disk is permanently destroyed. Explain which executed checks still apply and which require a separate restore experiment.

Observation: no success response inside the caller's deadline.
Known decision: the pre-crash acknowledged value was read after recovery.
Reconciliation: inspect unknown_write_after_recovery, not a guess.
Limit: this run did not destroy storage or partition packets.

For a next experiment, add backup, deliberately remove only your disposable cluster data, restore to new owned directories, and test both contents and client epochs. The first edition provides this as a design exercise, not an executed result.

Self-check

Read the originals and continue

• etcd disaster recovery, snapshots, restore and revision differences.
• Raft §7, log compaction and snapshots.
• MIT 6.5840, implementation and verification extensions.

Continue with MySQL Lab for transactions and local lock evidence, Streaming Lab for end-to-end state/effect boundaries, Distributed Compute Lab for recomputation and retries, and Java Runtime Lab for interruption and caller cancellation. Those are distinct layers; none automatically supplies this chapter's distributed guarantee.