RPC, unknown outcomes & retries

Learning goals

Identify unknown outcomes, design a retry identity, and atomically couple deduplication to a side effect.

Prerequisites: ordered state-machine decisions, request/response calls and atomic compare/update.

The ambiguity window

Suppose an increment commits at the server at 20 ms. The caller stops waiting at 150 ms. The response is deliberately held until 800 ms. Retrying a plain increment can apply the intended change twice. Consensus does not prevent this: the two increments are two perfectly valid commands in its log.

caller               application              etcd
  increment ───────► transaction ────────────► commit
  deadline 150 ms          │
  retry ───────────►      response held 800 ms
Without identity: another effect. With identity: stored result.

Idempotence means applying the same operation repeatedly has the same effect as applying it once. Setting a value to 7 can be idempotent; adding 7 generally is not. A retry key lets a service turn a repeated nonidempotent intent into a lookup of a previously committed result. Scope that key to the principal and operation, bind it to the payload, and define its retention and expiry behavior.

Implementation: effect and evidence share a commit

The toy HTTP application reads the counter, then submits one etcd Txn that compares the counter revision and, in guarded mode, the absence of the operation key. On success it writes both the new counter and the operation's amount/result. A concurrent change makes the compare fail; the application rereads and retries, up to a bounded limit.

if operation_key exists:
    require stored amount == requested amount
    return stored result
Txn:
    compare counter.mod_revision == observed_revision
    compare operation_key.version == 0
    put counter = old + amount
    put operation_key = {amount, result}

The durable operation key closes the gap between “effect happened” and “dedup record exists.” Putting the marker in a separate transaction would reopen that gap. Our effect is entirely inside etcd. Sending an email, charging a remote account or writing an unrelated database would need an additional protocol; this example does not claim exactly-once external effects.

Real experiment B: commit, delay, retry

The application commits before deliberately delaying its HTTP response by 800 ms; the caller's deadline is 150 ms. An independent normal request then retries the same operation. In plain mode the final counter must be 2; in guarded mode it must be 1. Four additional concurrent retries in guarded mode must return the saved result. Reusing the ID with amount 9 must return a conflict.

See the client outcome, server commit records and final counter. The server-side commit is evidence available to the experiment observer; it was not information available to the timed-out caller. The fault is an actual delayed response in a real HTTP server, not a generated diagram.

Hands-on: modify lab/services.py in a downloaded copy to write the dedup marker separately after the counter. Insert a crash between the writes and predict how a retry can duplicate the effect. Do this only in the owned learning runtime. Keep the baseline passing result for comparison. No dedup records expire in the baseline; production cleanup requires an explicit retry horizon and replay contract.

Self-check

Read the originals

• Raft §8, client retries and duplicate command handling.
• etcd JSON gateway, real Txn wire format.
• etcd operation completion definition, uncertainty after timeout.