A webhook or endpoint pool routes events to one or more registered destinations under shared delivery policy. It can support failover endpoints, regional destinations, event filters, fan-out, customer-specific URLs, internal service hooks, and agent callbacks. The difficulty is that HTTP delivery is not exactly once. Connections fail, responses disappear, endpoints time out, events arrive more than once, and destinations process messages out of order.
The reliable contract is therefore at-least-once delivery with stable event identifiers, authenticated messages, idempotent consumers, bounded retries, and replay tools. A pool adds selection and policy on top: which endpoint is eligible, whether to fan out or choose one, how to isolate tenant failures, and when to move an event to a dead-letter state.
Build an endpoint registry
Store policy with each destination
Each endpoint record needs a durable identifier, tenant, URL, enabled state, event subscriptions, content format, signing secret reference, timeout, retry policy, region, health state, and ownership metadata. Store secrets in a secret manager and return only masked references through ordinary reads. URL changes should create a revision or audit event because they redirect sensitive data.
Validate endpoints before activation. A challenge-response flow can prove control of the destination. Enforce HTTPS for external traffic, block private or link-local addresses unless explicitly allowed, and protect against DNS rebinding and server-side request forgery. Re-resolve and validate destinations according to policy rather than trusting the registration-time IP forever.
Use a stable event envelope
An event should include a globally unique identifier, type, version, occurrence time, producer, tenant context, subject, data payload, and trace identifier. Keep the envelope stable while versioning the payload schema. An event identifier represents the business occurrence; each delivery attempt gets a separate attempt identifier. This distinction lets consumers deduplicate the event while operators inspect every retry.
Do not place secrets in the payload. Minimize personal data and support field-level redaction by subscription. Large payloads can be represented by a short-lived signed reference, but the consumer must be able to retrieve it within the replay window. If the reference expires before retries end, regenerate it without changing the event identity.
Sign the bytes you send
Calculate a signature over the exact transmitted body plus a timestamp and, optionally, endpoint identifier. Include the signature version and key identifier in headers. The consumer verifies the timestamp window, selects the correct key, computes the signature, and uses a constant-time comparison. Signing a parsed object instead of the bytes can fail when serialization changes whitespace or key order.
Support secret rotation with overlapping keys. An endpoint can accept the current and next key during a transition. Record which key signed each attempt. Never include the secret itself in a test event or dashboard. Provide example verification code, but make the protocol specification—not a single SDK—the source of truth.
Define success and timeout precisely
A delivery attempt starts when the worker sends the request and ends with a response, network error, or timeout. Define which status codes are successful. Most systems treat a timely 2xx as acceptance. Redirects should usually not be followed automatically because they can bypass destination validation. A timeout is unknown, not proof that the consumer did nothing; the consumer may have processed the event before the response was lost.
Stripe’s official webhook documentation notes that events can be retried and may not arrive in generated order, illustrating why receivers must track event identifiers instead of relying on a single delivery or timestamp. A pool should make the same expectations explicit and expose delivery history to endpoint owners.
Retry with backoff, jitter, and limits
Cap retries and retain evidence
Retry transient failures with exponential backoff and jitter so a recovering endpoint is not flooded at once. Set a maximum attempt count or delivery window. Differentiate retryable network errors and 5xx responses from permanent failures such as an invalid URL or explicit 4xx rejection. Some 4xx responses may be temporary by contract, but do not retry every client error blindly.
Honor a reasonable Retry-After header where policy allows, while enforcing a maximum. Isolate queues by tenant or endpoint class so one failing customer cannot consume every worker. Use circuit breakers to pause an endpoint after repeated failure, then probe it before resuming full traffic. Surface the pause reason and next probe time.
Do not promise global ordering casually
Parallel workers, retries, and endpoint failures make strict ordering expensive. If ordering matters, define the scope: per subject, per account, or per stream. Attach a sequence number within that scope and let the consumer buffer or reject gaps. Strict serial delivery can let one stuck event block everything behind it, so use it only where the business operation requires it.
Consumers should be idempotent. Store processed event identifiers and make side effects conditional. When an event updates a versioned resource, compare resource version rather than arrival time. A later state event may arrive before an earlier one; applying both blindly can move the consumer backward.
Choose fan-out, active-passive, or eligible-one routing
Fan-out sends an event to every subscribed endpoint and tracks each delivery independently. Active-passive sends to a primary and uses a backup only under explicit conditions. Eligible-one routing chooses one endpoint based on region, health, capability, or load. These modes have different semantics and should be declared on the subscription.
Failover can duplicate side effects. If the primary processed the event but timed out, sending to a backup may process it again in a different system. Use a shared event identifier and require idempotency across failover destinations, or limit failover to events whose downstream action is safe. For command-like operations, a pull queue or transactional API may be better than webhooks.
Provide replay without creating a second event
Replay should create new delivery attempts for the same event identifier. Operators can replay one event, a time range, an event type, or all failed events for an endpoint. Require scope and confirmation for broad replays. Apply current or historical payload policy deliberately; replaying an old event with a new schema can surprise consumers.
Keep a retention policy for payloads and delivery logs. If payload content expires, the system may retain metadata without being able to replay. State that clearly in the API. For sensitive events, shorter retention and customer-triggered deletion may be appropriate. A replay log records the actor, filter, time, and resulting attempts.
Use a dead-letter state as an operational queue
After the retry budget ends, move the delivery to a terminal failed or dead-letter state. This is not a trash can. It should have a reason, last response summary, endpoint state, payload availability, and recommended action. Endpoint owners can fix the destination and replay. Platform operators can detect systemic issues such as a signing change or DNS failure across many tenants.
Alerting should be proportional. One failed test event may not page anyone. A high failure ratio, sustained queue age, or failure across several endpoints may. Define service objectives around delivery acceptance within a time window and replay availability, while acknowledging that the destination controls final processing.
A webhook pool surface
POST /endpoints
POST /endpoints/{id}/verify
PATCH /endpoints/{id}
POST /endpoints/{id}/rotate-secret
GET /events/{id}
GET /events/{id}/deliveries
POST /deliveries/{id}/replay
POST /replays
GET /dead-letter-deliveriesList endpoints and delivery logs with cursor pagination. Filter by event type, state, time, endpoint, and tenant. Mask URLs according to role when they contain sensitive paths. Return response status, duration, attempt time, and a bounded response preview, not unlimited downstream content.
Trace every event through the pool
Track publish-to-first-attempt latency, acceptance latency, retry count, queue age, response class, timeout, signature version, circuit state, and dead-letter rate. Measure by endpoint class and tenant without putting event identifiers in metric labels. Logs can carry event and attempt identifiers for detailed investigation.
Propagate a trace context when the consumer supports it, but do not trust returned trace fields as authoritative. Protect dashboards from payload exposure. Endpoint owners should see their own event metadata and attempts; platform staff should use elevated access for payload inspection with audit and redaction.
Reliable webhooks admit uncertainty
A dependable webhook pool does not pretend delivery is exactly once or always ordered. It gives every event a stable identity, signs exact bytes, retries predictably, isolates failures, and makes replay and dead-letter operations visible. The endpoint registry and routing policy add the pool behavior, while idempotent consumers complete the contract.
See the Node & Endpoint Pools hub and the companion API portal design guide.
