Webhook Retries Without Duplicate Side Effects

Webhooks fail. Networks jitter, receivers deploy, and partners misconfigure URLs. Retries are necessary; duplicate side effects are not.

Our lab starts with signed payloads and a delivery ID stored before any business logic runs. Receivers acknowledge only after the inbox row commits. Handlers check the delivery ID before mutating state.

For long-running work, return 202 with a status URL rather than holding the connection open. Partners appreciate predictable timeouts.

We also run a chaos exercise: learners disable their database for two minutes and watch retry storms in a sandbox. The goal is a runbook, not heroics.