We shipped an idempotency bug that ran your side effects twice

Our idempotency layer had a check-then-act race, on a product whose whole job is to not do things twice. Here's the bug, the fix, the test that proves it, and the three related things we found while looking.