METHODS & TOOLS / INTERACTIVE EDITION · ETCD 3.6.6
Real experiment evidence
SAVED REAL EVIDENCE
Walk through a real run
Replay selected observations from all three saved repetitions.
Loading activity… Enable JavaScript to interact; the course text remains readable.
Ordered summaries from the saved JSON, not a packet-level replay. Read the concept · Inspect real evidence
Recorded results from real processes. All three repetitions remain inspectable.
REAL PROCESS EVIDENCE / 2026-11-13T17:39:41.195190+00:00
Topology and conditions
etcd 3.6.6 · Darwin arm64 · Python 3.11.2
Three loopback voting processes, separate WAL directories, one host. Actual process kill/pause and delayed HTTP responses. No packet partition, storage destruction or multi-host performance measurement. Durations use the orchestrator’s monotonic clock and include request round trips.
SHA-256: df927af381c415e574f1a48755345be7a1bcaee243fdbc3f73002a05ab6bbbf6
| Repeat | Leader | 1 down | 2 down write/read | plain / dedup | Fence tokens |
|---|---|---|---|---|---|
| 1 | n1 → n2 | ok | unknown / unknown | 2 / 1 | 9 < 11 |
| 2 | n1 → n3 | ok | unknown / unknown | 2 / 1 | 9 < 11 |
| 3 | n1 → n3 | ok | unknown / unknown | 2 / 1 | 9 < 11 |
A · Leader crash and quorum loss
The acknowledged pre-crash value survives and the restarted member catches up. One unavailable follower still permits a successful write. With two followers paused, both write and default read return no success inside their deadlines. Inspect the eventual value of the unknown write separately.
REAL RUN / 01
n1 → n2; 1 down: ok; 2 down: unknown; reconciled = maybe-committed
Inspect complete exported record
{
"before": {
"leader": "n1",
"revision": 2,
"term": 2
},
"new_leader": "n2",
"committed_write_survived": "acknowledged-before-crash",
"failover_write": {
"outcome": "ok",
"duration_ms": 11.609,
"response": {
"header": {
"cluster_id": "17193519525117484670",
"member_id": "9419852838338823221",
"revision": "3",
"raft_term": "3"
}
}
},
"wal_restart": {
"node": "n1",
"value": "acknowledged-before-crash",
"after_crash_value": "committed"
},
"one_unreachable": {
"outcome": "ok",
"duration_ms": 13.268,
"response": {
"header": {
"cluster_id": "17193519525117484670",
"member_id": "9419852838338823221",
"revision": "4",
"raft_term": "3"
}
}
},
"two_unreachable_write": {
"outcome": "unknown",
"duration_ms": 702.506
},
"two_unreachable_linearizable_read": {
"outcome": "unknown",
"duration_ms": 701.85
},
"unknown_write_after_recovery": "maybe-committed",
"conditions": {
"nodes": 3,
"quorum": 2,
"faults": [
"leader SIGKILL and same-data-dir restart",
"one follower SIGSTOP",
"two followers SIGSTOP"
],
"election_timeout_ms": 1000,
"client_deadline_ms": 700
},
"assertions_passed": true
}REAL RUN / 02
n1 → n3; 1 down: ok; 2 down: unknown; reconciled = maybe-committed
Inspect complete exported record
{
"before": {
"leader": "n1",
"revision": 2,
"term": 2
},
"new_leader": "n3",
"committed_write_survived": "acknowledged-before-crash",
"failover_write": {
"outcome": "ok",
"duration_ms": 10.696,
"response": {
"header": {
"cluster_id": "4613693674493055459",
"member_id": "10308100773522122874",
"revision": "3",
"raft_term": "3"
}
}
},
"wal_restart": {
"node": "n1",
"value": "acknowledged-before-crash",
"after_crash_value": "committed"
},
"one_unreachable": {
"outcome": "ok",
"duration_ms": 16.954,
"response": {
"header": {
"cluster_id": "4613693674493055459",
"member_id": "10308100773522122874",
"revision": "4",
"raft_term": "3"
}
}
},
"two_unreachable_write": {
"outcome": "unknown",
"duration_ms": 702.975
},
"two_unreachable_linearizable_read": {
"outcome": "unknown",
"duration_ms": 702.011
},
"unknown_write_after_recovery": "maybe-committed",
"conditions": {
"nodes": 3,
"quorum": 2,
"faults": [
"leader SIGKILL and same-data-dir restart",
"one follower SIGSTOP",
"two followers SIGSTOP"
],
"election_timeout_ms": 1000,
"client_deadline_ms": 700
},
"assertions_passed": true
}REAL RUN / 03
n1 → n3; 1 down: ok; 2 down: unknown; reconciled = maybe-committed
Inspect complete exported record
{
"before": {
"leader": "n1",
"revision": 2,
"term": 2
},
"new_leader": "n3",
"committed_write_survived": "acknowledged-before-crash",
"failover_write": {
"outcome": "ok",
"duration_ms": 12.79,
"response": {
"header": {
"cluster_id": "10587831175002086049",
"member_id": "15120877559436342554",
"revision": "3",
"raft_term": "3"
}
}
},
"wal_restart": {
"node": "n1",
"value": "acknowledged-before-crash",
"after_crash_value": "committed"
},
"one_unreachable": {
"outcome": "ok",
"duration_ms": 11.273,
"response": {
"header": {
"cluster_id": "10587831175002086049",
"member_id": "15120877559436342554",
"revision": "4",
"raft_term": "3"
}
}
},
"two_unreachable_write": {
"outcome": "unknown",
"duration_ms": 702.407
},
"two_unreachable_linearizable_read": {
"outcome": "unknown",
"duration_ms": 702.367
},
"unknown_write_after_recovery": "maybe-committed",
"conditions": {
"nodes": 3,
"quorum": 2,
"faults": [
"leader SIGKILL and same-data-dir restart",
"one follower SIGSTOP",
"two followers SIGSTOP"
],
"election_timeout_ms": 1000,
"client_deadline_ms": 700
},
"assertions_passed": true
}B · Timeout after commit and deduplication
Each first request commits, then its response is delayed. Both callers time out. A retry causes two effects without deduplication, one with the atomic operation key. Concurrent replays and changed-payload conflicts are included.
REAL RUN / 01
2 → 1 (plain / dedup)
Inspect complete exported record
{
"conditions": {
"response_delay_after_commit_ms": 800,
"client_deadline_ms": 150,
"effect": "etcd CAS counter plus durable operation-result key in one Txn",
"dedup_retention": "no expiry in this bounded experiment"
},
"cases": [
{
"dedup": false,
"first_client": {
"outcome": "unknown",
"duration_ms": 151.713
},
"commits_before_delayed_response": [
{
"kind": "commit",
"id": "synthetic-op-1",
"value": 1,
"revision": 6
}
],
"retry_response": {
"amount": 1,
"value": 2,
"replayed": false
},
"final_counter": 2,
"commits": [
{
"kind": "commit",
"id": "synthetic-op-1",
"value": 1,
"revision": 6
},
{
"kind": "commit",
"id": "synthetic-op-1",
"value": 2,
"revision": 7
}
]
},
{
"dedup": true,
"first_client": {
"outcome": "unknown",
"duration_ms": 151.561
},
"commits_before_delayed_response": [
{
"kind": "commit",
"id": "synthetic-op-1",
"value": 1,
"revision": 8
}
],
"retry_response": {
"amount": 1,
"value": 1,
"replayed": true
},
"final_counter": 1,
"commits": [
{
"kind": "commit",
"id": "synthetic-op-1",
"value": 1,
"revision": 8
}
],
"concurrent_retries": [
{
"amount": 1,
"value": 1,
"replayed": true
},
{
"amount": 1,
"value": 1,
"replayed": true
},
{
"amount": 1,
"value": 1,
"replayed": true
},
{
"amount": 1,
"value": 1,
"replayed": true
}
],
"changed_payload": {
"error": "id-conflict"
}
}
],
"assertions_passed": true
}REAL RUN / 02
2 → 1 (plain / dedup)
Inspect complete exported record
{
"conditions": {
"response_delay_after_commit_ms": 800,
"client_deadline_ms": 150,
"effect": "etcd CAS counter plus durable operation-result key in one Txn",
"dedup_retention": "no expiry in this bounded experiment"
},
"cases": [
{
"dedup": false,
"first_client": {
"outcome": "unknown",
"duration_ms": 151.176
},
"commits_before_delayed_response": [
{
"kind": "commit",
"id": "synthetic-op-1",
"value": 1,
"revision": 6
}
],
"retry_response": {
"amount": 1,
"value": 2,
"replayed": false
},
"final_counter": 2,
"commits": [
{
"kind": "commit",
"id": "synthetic-op-1",
"value": 1,
"revision": 6
},
{
"kind": "commit",
"id": "synthetic-op-1",
"value": 2,
"revision": 7
}
]
},
{
"dedup": true,
"first_client": {
"outcome": "unknown",
"duration_ms": 152.061
},
"commits_before_delayed_response": [
{
"kind": "commit",
"id": "synthetic-op-1",
"value": 1,
"revision": 8
}
],
"retry_response": {
"amount": 1,
"value": 1,
"replayed": true
},
"final_counter": 1,
"commits": [
{
"kind": "commit",
"id": "synthetic-op-1",
"value": 1,
"revision": 8
}
],
"concurrent_retries": [
{
"amount": 1,
"value": 1,
"replayed": true
},
{
"amount": 1,
"value": 1,
"replayed": true
},
{
"amount": 1,
"value": 1,
"replayed": true
},
{
"amount": 1,
"value": 1,
"replayed": true
}
],
"changed_payload": {
"error": "id-conflict"
}
}
],
"assertions_passed": true
}REAL RUN / 03
2 → 1 (plain / dedup)
Inspect complete exported record
{
"conditions": {
"response_delay_after_commit_ms": 800,
"client_deadline_ms": 150,
"effect": "etcd CAS counter plus durable operation-result key in one Txn",
"dedup_retention": "no expiry in this bounded experiment"
},
"cases": [
{
"dedup": false,
"first_client": {
"outcome": "unknown",
"duration_ms": 151.671
},
"commits_before_delayed_response": [
{
"kind": "commit",
"id": "synthetic-op-1",
"value": 1,
"revision": 6
}
],
"retry_response": {
"amount": 1,
"value": 2,
"replayed": false
},
"final_counter": 2,
"commits": [
{
"kind": "commit",
"id": "synthetic-op-1",
"value": 1,
"revision": 6
},
{
"kind": "commit",
"id": "synthetic-op-1",
"value": 2,
"revision": 7
}
]
},
{
"dedup": true,
"first_client": {
"outcome": "unknown",
"duration_ms": 152.451
},
"commits_before_delayed_response": [
{
"kind": "commit",
"id": "synthetic-op-1",
"value": 1,
"revision": 8
}
],
"retry_response": {
"amount": 1,
"value": 1,
"replayed": true
},
"final_counter": 1,
"commits": [
{
"kind": "commit",
"id": "synthetic-op-1",
"value": 1,
"revision": 8
}
],
"concurrent_retries": [
{
"amount": 1,
"value": 1,
"replayed": true
},
{
"amount": 1,
"value": 1,
"replayed": true
},
{
"amount": 1,
"value": 1,
"replayed": true
},
{
"amount": 1,
"value": 1,
"replayed": true
}
],
"changed_payload": {
"error": "id-conflict"
}
}
],
"assertions_passed": true
}C · Lease expiry and stale writer
A is paused until the leased lock is observed absent. B obtains a larger create_revision and writes first. A resumes: plain mode accepts its stale value; fenced mode rejects it and retains B-current.
REAL RUN / 01
tokens 9 < 11; plain = A-late; fenced = B-current
Inspect complete exported record
{
"conditions": {
"fault": "SIGSTOP owned worker A; no lease keepalive",
"sink": "HTTP plus SQLite BEGIN IMMEDIATE; token check and write atomic"
},
"A": {
"owner": "A",
"token": 9,
"lease": "136130354206351129",
"requested_ttl_seconds": 2,
"granted_ttl_seconds": 2
},
"B": {
"owner": "B",
"token": 11,
"lease": "136130354206351154",
"requested_ttl_seconds": 2,
"granted_ttl_seconds": 2
},
"observed_key_absence_after_ms": 2063.627,
"B_writes": {
"plain": {
"accepted": true,
"stored_token": 11,
"stored_value": "B-current"
},
"fenced": {
"accepted": true,
"stored_token": 11,
"stored_value": "B-current"
}
},
"A_late_writes": {
"plain": {
"accepted": true,
"stored_token": 9,
"stored_value": "A-late"
},
"fenced": {
"accepted": false,
"stored_token": 11,
"stored_value": "B-current"
}
},
"assertions_passed": true
}REAL RUN / 02
tokens 9 < 11; plain = A-late; fenced = B-current
Inspect complete exported record
{
"conditions": {
"fault": "SIGSTOP owned worker A; no lease keepalive",
"sink": "HTTP plus SQLite BEGIN IMMEDIATE; token check and write atomic"
},
"A": {
"owner": "A",
"token": 9,
"lease": "4664218304555072282",
"requested_ttl_seconds": 2,
"granted_ttl_seconds": 2
},
"B": {
"owner": "B",
"token": 11,
"lease": "4664218304555072308",
"requested_ttl_seconds": 2,
"granted_ttl_seconds": 2
},
"observed_key_absence_after_ms": 2186.343,
"B_writes": {
"plain": {
"accepted": true,
"stored_token": 11,
"stored_value": "B-current"
},
"fenced": {
"accepted": true,
"stored_token": 11,
"stored_value": "B-current"
}
},
"A_late_writes": {
"plain": {
"accepted": true,
"stored_token": 9,
"stored_value": "A-late"
},
"fenced": {
"accepted": false,
"stored_token": 11,
"stored_value": "B-current"
}
},
"assertions_passed": true
}REAL RUN / 03
tokens 9 < 11; plain = A-late; fenced = B-current
Inspect complete exported record
{
"conditions": {
"fault": "SIGSTOP owned worker A; no lease keepalive",
"sink": "HTTP plus SQLite BEGIN IMMEDIATE; token check and write atomic"
},
"A": {
"owner": "A",
"token": 9,
"lease": "7099539803060749087",
"requested_ttl_seconds": 2,
"granted_ttl_seconds": 2
},
"B": {
"owner": "B",
"token": 11,
"lease": "7099539803060749113",
"requested_ttl_seconds": 2,
"granted_ttl_seconds": 2
},
"observed_key_absence_after_ms": 2181.863,
"B_writes": {
"plain": {
"accepted": true,
"stored_token": 11,
"stored_value": "B-current"
},
"fenced": {
"accepted": true,
"stored_token": 11,
"stored_value": "B-current"
}
},
"A_late_writes": {
"plain": {
"accepted": true,
"stored_token": 9,
"stored_value": "A-late"
},
"fenced": {
"accepted": false,
"stored_token": 11,
"stored_value": "B-current"
}
},
"assertions_passed": true
}What remains unmeasured
These checks establish the asserted outcomes in this bounded schedule, not universal correctness or production capacity. Recovery uses the same durable directory; backup restore is not executed. The history workbench uses synthetic fixtures and does not certify these records. Raw logs remain local; this report contains only selected synthetic observations.