“I did not receive success” and “the operation did not happen” are different statements.
Identify unknown outcomes, design a retry identity, and atomically couple deduplication to a side effect.
Prerequisites: ordered state-machine decisions, request/response calls and atomic compare/update.
Suppose an increment commits at the server at 20 ms. The caller stops waiting at 150 ms. The response is deliberately held until 800 ms. Retrying a plain increment can apply the intended change twice. Consensus does not prevent this: the two increments are two perfectly valid commands in its log.
caller application etcd increment ───────► transaction ────────────► commit deadline 150 ms │ retry ───────────► response held 800 ms Without identity: another effect. With identity: stored result.
Idempotence means applying the same operation repeatedly has the same effect as applying it once. Setting a value to 7 can be idempotent; adding 7 generally is not. A retry key lets a service turn a repeated nonidempotent intent into a lookup of a previously committed result. Scope that key to the principal and operation, bind it to the payload, and define its retention and expiry behavior.
The toy HTTP application reads the counter, then submits one etcd Txn that compares the counter revision and, in guarded mode, the absence of the operation key. On success it writes both the new counter and the operation's amount/result. A concurrent change makes the compare fail; the application rereads and retries, up to a bounded limit.
if operation_key exists:
require stored amount == requested amount
return stored result
Txn:
compare counter.mod_revision == observed_revision
compare operation_key.version == 0
put counter = old + amount
put operation_key = {amount, result}The durable operation key closes the gap between “effect happened” and “dedup record exists.” Putting the marker in a separate transaction would reopen that gap. Our effect is entirely inside etcd. Sending an email, charging a remote account or writing an unrelated database would need an additional protocol; this example does not claim exactly-once external effects.
The application commits before deliberately delaying its HTTP response by 800 ms; the caller's deadline is 150 ms. An independent normal request then retries the same operation. In plain mode the final counter must be 2; in guarded mode it must be 1. Four additional concurrent retries in guarded mode must return the saved result. Reusing the ID with amount 9 must return a conflict.
See the client outcome, server commit records and final counter. The server-side commit is evidence available to the experiment observer; it was not information available to the timed-out caller. The fault is an actual delayed response in a real HTTP server, not a generated diagram.
Hands-on: modify lab/services.py in a downloaded copy to write the dedup marker separately after the counter. Insert a crash between the writes and predict how a retry can duplicate the effect. Do this only in the owned learning runtime. Keep the baseline passing result for comparison. No dedup records expire in the baseline; production cleanup requires an explicit retry horizon and replay contract.
It orders both commands; it does not know they represent the same business intent. The application needs an identity and atomic effect/result recording.
No. The baseline detects that conflict. Silently replaying an unrelated old result would hide a caller error.