Reliable Webhooks: Signatures, Retries, Idempotency, and Ordering
Build dependable webhook integrations with signature verification, replay protection, idempotent handlers, retries, queues, ordering, and observability.
On this page
A webhook lets one system notify another when an event occurs. The receiver must assume that the network can delay, duplicate, reorder, or lose messages. Reliable webhook handling looks like an at-least-once message system.
#Define an event contract
Give every event a stable identifier, event type, creation time, schema version, and payload. Document whether a payload is a snapshot or a reference to current state and which fields are immutable.
#Verify signatures
Sign the raw request body with a secret known to sender and receiver. Verify the signature before parsing or acting. Include a timestamp in the signed material to reduce replay risk, rotate secrets with overlap, and never log signature secrets or bearer URLs.
#Persist and process asynchronously
Verify and persist the event, then respond quickly. Put business work on a durable queue or job table. Do not acknowledge a message that was not safely persisted. Use timeouts and a dead-letter path for failures that need investigation.
#Idempotency and ordering
Store the event identifier with a unique database constraint so duplicate deliveries do not repeat side effects. For payments, emails, and provisioning, use a downstream idempotency key as well.
Events can arrive out of order. Include a sequence or domain version and ignore older updates when a newer version has already been applied.
#Retries and observability
Retry temporary failures with exponential backoff and a limit. Distinguish invalid input, unauthorized requests, and temporary dependency failures. Log event ID, attempt, type, tenant, response status, duration, and outcome while redacting sensitive payloads.
#Testing webhooks
Replay the same event and confirm no duplicate effect.
Change one byte and confirm signature failure.
Reject expired timestamps.
Deliver events out of order.
Simulate timeouts and downstream outages.
Test oversized and malformed payloads.
#Common mistakes
Parsing before verifying the signature.
Returning success before persistence.
Assuming one delivery per event.
Doing long work inside the request.
Having no replay or dead-letter process.
#Frequently asked questions
#Are webhooks guaranteed once?
No. Design for duplicates, delay, reordering, and eventual failure.
#How do I prevent duplicate processing?
Persist a unique event or operation identifier and make downstream effects idempotent.
#Conclusion
Webhooks become dependable when treated as distributed messages rather than ordinary HTTP requests. Verify, persist, process asynchronously, make effects idempotent, handle ordering, and monitor every delivery.