Leases, locks & fencing

Learning goals

Distinguish lock ownership from downstream authority; construct a monotonically increasing fencing token; locate the atomic enforcement boundary.

Prerequisites: unknown outcomes and atomic updates, leases and process pauses.

The stale holder is still a process

Worker A acquires a leased lock, reads some data and pauses. The lease expires at the coordinator. B acquires the lock and writes a newer result. A resumes with its old local belief and sends a write. An ordinary downstream service that never consults a token can accept A's stale write after B's fresh one. The lock service itself need not be incorrect.

A: acquire token 12 ── PAUSE ───────────────── resume → old write
etcd: lock exists ─── expiry ── new lock token 14
B:                            acquire ── write new value
sink:                          observe 14 ── reject 12

A fencing token orders ownership generations. The receiving resource persists the highest accepted generation and atomically checks it together with the protected update. A token lower than the high-water mark is rejected. This does not cancel a paused process; it limits which late requests the resource accepts.

Concrete implementation

The baseline uses an etcd Txn comparing VERSION(lock)=0 and creates one key attached to a two-second lease. The key's create_revision is the token, not the lease ID. With the same ongoing cluster lineage, later successful creations have larger revisions. The sink uses a SQLite BEGIN IMMEDIATE transaction: read high-water mark, compare token, update value and mark together.

Within one ownership generation the baseline accepts equal tokens, so fencing does not deduplicate the owner's repeated writes or order its own concurrent commands. It rejects old generations only after observing a newer token. If A arrives after expiry but before B presents its token, a high-water-mark-only sink can still accept A. Stronger current-ownership semantics need a different protocol or handoff boundary.

Snapshot restore or a replacement cluster can break a naïve revision assumption. Recovery must preserve a strictly newer epoch or use a rigorously managed revision strategy before reusing the token space. The downstream mark must also remain durable and all relevant write paths must enforce it. A client-side check followed by a separate write is not an atomic fence.

Real experiment C: the late write

A is a separate owned Python process, actually paused with SIGSTOP. No lease keepalive is sent. The runner polls for the lock key's observed absence, instead of assuming exactly two seconds implies expiry. B then acquires a higher token and writes to two modes of the same HTTP/SQLite sink. The runner resumes A and it submits its old token to both modes.

The plain mode must end at A-late. The fenced mode must reject A and remain at B-current. Inspect the actual tokens, observed expiry wait and both responses. These are real coordinator and downstream requests; the diagram above uses illustrative token numbers.

Hands-on: locate the comparison in lab/services.py. Explain why moving it out of the SQLite transaction permits a race. As an extension, have A write before B has contacted the sink; state why the high-water mark may accept A without violating the narrower contract demonstrated here.

Self-check

Read the originals

• Burrows — The Chubby lock service, §2.4 on sequencers and delayed requests.
• etcd API reference, lease TTL and create_revision fields.
• etcd recovery, revision differences after restore.