An idempotency key exists to make a retry safe. You send the same key twice, the work happens once. That is the entire contract, and on a platform that delivers messages and invokes tools on your behalf, it is close to the only promise that matters.
Ours was broken. Under concurrency, two requests carrying the same idempotency key could both run the handler, and both commit their side effects.
This post is what the bug was, why the obvious implementation is wrong, how we fixed it, and the three adjacent problems we only found because we went looking. We are writing it up because "we had a bug in the thing you pay us to get right" is worth more said out loud than quietly patched.
The bug
The original implementation did the natural thing:
1. read the cached response for this key
2. if present -> return it
3. run the handler
4. write the result to the cache
Read the code path with one request in your head and it is correct. Read it with two and the problem is immediate: steps 1 and 4 are not atomic with respect to each other.
Two concurrent requests arrive with the same key. Both reach step 1. Neither finds a cached response, because neither has finished yet. Both fall through to step 3. Both run the handler. Both commit.
This is a check-then-act race, which is one of the oldest bugs there is, and it is easy to ship precisely because it is invisible at low traffic. Sequential retries, the kind you produce by hand while testing, work perfectly. The cache is populated by the time the second request arrives, and the second request replays. You need genuine overlap to see it, and genuine overlap is exactly what you get from an HTTP client's automatic retry on a slow response, which is the single most common reason an idempotency key exists in the first place.
The failure mode is therefore worst under precisely the conditions the feature was built for.
The fix: claim, don't check
The correct shape is to stop treating the cache as something you read, and start treating the key as something you claim. A claim has to be atomic, and Redis gives you that directly:
1. SET key <claim-token> NX -- atomically claim, or fail
2. if the claim failed -> someone else owns this key; wait, then replay their result
3. run the handler
4. compare-and-set the result against our claim token
SET NX succeeds for exactly one caller. That single atomic operation is the whole difference: instead of two requests asking "has this happened yet?" and both hearing no, one request asserts "this is mine" and the other is told it is not.
Everything else is consequence:
- Same request, replayed: returns the stored response with
X-Idempotent-Replay: true. - Same key, different payload: returns
409 idempotency_key_reuserather than silently serving someone else's result. Reusing a key for different work is a bug in the caller and it should be loud. - Overlapping duplicate: waits for the in-flight owner, then replays its result.
- Handler failed: releases the claim, so a retry after a genuine error can proceed instead of being locked out by a claim nobody will ever complete.
That last case is where the claim token earns its keep. Completion and release are guarded by a Lua compare-and-set against the token, so a request can only complete or release its own claim. Without it, a slow request that already lost its lease could return late and overwrite a newer request's result. You would have fixed the race and introduced a subtler one.
Two more edges we got wrong on the first pass and fixed:
- Body fingerprinting is JSON-only. We fingerprint the request body to detect key reuse with a different payload. Doing that to a multipart upload consumes the stream, which breaks file uploads in an entirely new way. Uploads are fingerprinted by key alone.
- Oversized successful responses refuse to replay rather than re-run. If a 2xx response is too large to store, the honest behaviour is to fail the replay loudly. Quietly re-running the handler would be the original bug wearing a hat.
Pending claims carry a self-healing five-minute lease, so a process that dies mid-handler does not wedge a key forever. Results are retained for 24 hours.
The test is the point
The fix is a dozen lines. The test is the deliverable.
We wrote a test that fires five genuinely concurrent requests carrying the same idempotency key at a real Redis instance, and counts how many times the handler body executes.
Against the old implementation, the handler runs five times. Against the fix, it runs once.
One property makes it a regression test rather than a decoration: it fails against the old code and passes against the new. A test that passes against both implementations tells you nothing about the bug. It only tells you it was written after the fix. We ran ours against the pre-fix code deliberately, to watch it fail, before trusting it.
It also has to run against real Redis. A mocked Redis is a model of Redis written by the person who misunderstood the concurrency semantics in the first place, and it will happily agree with them. Our backend integration tests skip when Postgres and Redis are absent, and a skipped test is not verification. It is a green tick with nothing behind it.
Three things we found while we were in there
Fixing the race meant auditing every claim we made about idempotency. None of these were the reported bug. All of them were real.
The TypeScript SDK supported idempotency keys on zero operations. Not "some". Zero. The Python SDK supported five of nine resource groups. The documentation described a uniform feature across both. Our own contract gates now assert that every operation marked x-anlyon-idempotency: supported accepts a key in both SDKs, and adding a new one fails CI until both are implemented.
Both SDKs discarded headers on PATCH and DELETE. The base resource classes accepted an options object and dropped it on those two verbs. So the calls that did the most destructive work were the ones least able to be made safe.
Fourteen methods accepted an idempotency key and silently threw it away. This is the one that changed how we work. We had a manifest declaring which operations supported idempotency, and the manifest was green. But a manifest records a claim, not a behaviour. When we added a test asserting the header actually reached the wire, fourteen methods failed it. They took the argument, satisfied the type checker, and dropped it before the request was built.
The rule we took from that: when you add a capability to a manifest, add a test that asserts the behaviour reaches the wire. Otherwise the manifest is a list of things you intended.
Why write this up
Because the alternative is a changelog entry saying "improved idempotency handling", and because we would rather you trusted the thing we shipped after the audit than the thing we shipped before it.
There is also a general point in here, and it is the reason this bug bothered us more than most. Anlyon's argument is that agent side effects should be correct, isolated, governed, and recoverable: that the interesting failures in agent systems are not bad reasoning but unbounded action. Duplicate side effects under retry are exactly that failure, in our own code, in the layer specifically responsible for preventing it.
The gates that catch this class of thing now run on every commit: route coverage against the canonical contract, operation quality including the SDK and idempotency manifests, docs drift, and published-reference sync. They fail loudly and they never skip. That is a better guarantee than our good intentions, which is the point.