Classify Webhook Failures Before Retrying Them
Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.
A webhook event with an invalid amount and an event waiting for a temporarily unavailable dependency should not receive the same retry policy. Repeating the first wastes processing capacity. Abandoning the second too early can leave valid work unfinished. A third case—a request that timed out after possibly creating a record—may become more dangerous when repeated.
Retry errors only when there is evidence that another attempt could succeed and repeating the operation is safe. Hold events that need a specific prerequisite, stop events that cannot succeed unchanged, and investigate events whose outcome is uncertain. These are four different operating decisions: retryable, blocked, permanent, and unknown.
The practical goal is not a larger retry count. It is a recorded reason for every retry, an observable condition for releasing every hold, and a clear endpoint for every event that remains unresolved.
1. Separate retry eligibility from retry timing
A retry schedule answers when to try again. Failure classification answers whether trying again makes sense at all. Decide eligibility first; otherwise, an elaborate backoff schedule merely delays the same invalid operation.
Classify the failed processing step, not just the webhook envelope. One event may pass validation, update an internal record, and then fail while requesting an external action. Replaying the entire event can repeat work that already succeeded.
Before deciding, record three facts:
- Failure stage: parsing, validation, context lookup, authorisation, dependency call, or outcome confirmation.
- Last confirmed result: the last step known to have completed, with a durable reference where available.
- Side-effect boundary: whether the failed attempt could already have changed another system.
An HTTP response code alone is not enough. A missing-record response could mean the record has not arrived yet, the identifier is wrong, or the caller is looking in the wrong workspace. A timeout could occur before an action starts or after it succeeds. The classification needs the processing context as well as the technical symptom.
2. Define four categories with different release rules
Retryable: another attempt has a supported path to success
Use this category when the cause appears transient, the event remains valid, and repetition is safe. Evidence might include an explicit rate-limit response or a connection failure before any action was submitted. The next attempt should have a scheduled time and a bounded retry budget.
HTTP 429 indicates rate limiting, and a Retry-After header may specify when to retry. Providers differ in how they apply limits, so interpret that response using the dependency's documented behaviour rather than a universal timer. See the MDN explanation of HTTP 429.
Blocked: a named prerequisite must change
Use this category when a valid event cannot proceed until something specific happens: related context becomes available, access is restored, or a supported mapping is configured. Repeated attempts do not themselves fix that prerequisite.
A blocked event needs a release condition and an owner. It can have a limited check schedule, but checking whether a prerequisite exists is different from repeatedly attempting the business action.
Permanent: the unchanged event should stop
Use this category when the event is invalid, irrelevant to the supported workflow, or ineligible for the requested action. Another attempt with the same payload and processing rules has no supported path to success.
Permanent means terminal for that event version under the current rules. It does not mean the underlying business issue can never be corrected. Corrected input or an approved implementation change may justify a controlled reprocessing decision.
Unknown: evidence is insufficient to choose safely
Use this category when the cause or outcome is uncertain. A timeout after submitting a record-creation request is a typical example: failure to receive confirmation is not proof that creation failed.
Unknown is an investigation state, not a slower retry queue. Assign a reconciliation task or review deadline. Do not automatically repeat a potentially completed action merely because the retry budget has room.
3. Use this filled failure-classification table
Fictional worked example: A business receives webhook events that update account records and sometimes request a downstream task. Every event, identifier, timing choice, and limit below is hypothetical. These are recommended design choices, not claims about a particular product.
The compact table shows the routing decision. The row details immediately below supply the evidence and stopping rules needed to operate it.
Failure Category Immediate route Malformed amount Permanent Stop unchanged event Related account not yet present Blocked Await account context Dependency rate-limited Retryable Schedule safe retry Unsupported event type Permanent Record and stop Timeout after task submission Unknown Reconcile outcome
Row A: malformed data
- Observed evidence: The amount field contains text where the supported event contract requires a numeric value. Validation fails before any business action.
- Decision: Permanent for this payload. Store a clear validation reason; do not repeatedly submit it to the same validator.
- Release or recovery: Request corrected input through the agreed source process. Preserve the relationship between the original event and its correction.
- Stop condition: Automatic processing ends immediately. Reconsider only after the input or an incorrectly implemented validation rule changes.
Row B: temporarily missing context
- Observed evidence: A valid event references an account absent locally. An ingestion record shows that the related account import is still pending.
- Decision: Blocked, because there is a named missing prerequisite and evidence it may arrive.
- Release or recovery: Resume when the account is present and the relationship has been validated. In this hypothetical policy, check after ten minutes and send unresolved cases for review after two hours.
- Stop condition: At the review deadline, investigate the source relationship. Do not leave the event circulating because the account might eventually appear.
Row C: unavailable dependency
- Observed evidence: The dependency returns a rate-limit response. Its documented behaviour establishes that this request was not applied, and the operation has a verified duplicate-protection mechanism.
- Decision: Retryable. Respect a valid retry instruction and the dependency-specific policy.
- Release or recovery: Schedule a delayed attempt, retaining the same logical action identity.
- Stop condition: In this hypothetical policy, stop automatic attempts after five total attempts or thirty minutes, whichever comes first. Move the event to owned review rather than deleting it or retrying indefinitely.
Row D: unsupported event type
- Observed evidence: The payload is readable, but its event type is absent from the workflow's supported-type registry.
- Decision: Permanent under the current implementation. Distinguish expected, intentionally ignored types from unexpected unsupported types that deserve investigation.
- Release or recovery: Reprocess only after support is deliberately added, tested, and authorised for the relevant events.
- Stop condition: No scheduled retries. An unexpected rise in these events creates an integration-review task, not permission to execute them.
Row E: uncertain action outcome
- Observed evidence: The worker submitted a task-creation request, then timed out before recording a confirmed result. No reliable completion reference is available locally.
- Decision: Unknown. Neither success nor failure is established.
- Release or recovery: Search the downstream system using a supported correlation mechanism. A confirmed existing task resolves the event without creating another. A confirmed non-execution may permit a safe retry.
- Stop condition: In this hypothetical policy, unresolved cases reach a named reviewer after fifteen minutes. If absence cannot be established reliably, keep execution paused pending an explicit decision.
4. Require evidence before changing a classification
Classification should be revisable, but not arbitrary. Record what observation would move an event into another category.
A missing account moves from blocked to ready when the required account and relationship are verified. If the source confirms that the identifier is invalid, it becomes permanent instead. A rate-limited request may remain retryable until its budget expires; exhaustion ends automatic execution but does not prove that the payload is permanently invalid.
For each failure record, retain:
- Event identity, logical action identity, and relevant workspace or tenant scope.
- Processing stage and last confirmed completed step.
- Category, specific reason, and supporting observation.
- Attempt count, first failure time, and next permitted action.
- Release condition, expiry or review deadline, and responsible owner.
- Known side effects and any reconciliation result.
Keep diagnostic records proportionate. Store useful references and sanitised error details rather than copying sensitive payloads into every log entry.
Do not convert access failures into successful processing by relaxing permission checks. A changed credential may resolve a configuration problem, but the resumed action still requires authorisation. OWASP's authorisation guidance recommends least privilege, deny-by-default behaviour, and permission checks on every request.
5. Bound retries by attempts, age, and continuing relevance
A retry budget should limit both repeated work and elapsed waiting. An attempt limit alone can leave work pending too long when delays grow. An age limit alone can permit excessive rapid attempts.
Use delayed retries with spacing that increases after repeated transient failures, plus jitter where appropriate to avoid synchronising many events. These are scheduling techniques, not evidence that an event is eligible to retry.
Before each attempt, check that the action is still needed. An event can become obsolete while waiting: a newer state may supersede it, or the underlying request may have been withdrawn. Do not assume that eventual execution is always preferable to stopping.
Budget exhaustion should produce a distinct state such as retry_exhausted, with an owner and retained evidence. It should not silently discard the event, relabel it as invalid, or start a fresh budget under a new queue identity.
The trade-off is deliberate: shorter budgets reduce repeated load but send more recoverable work to review. Longer budgets allow more autonomous recovery but increase delay and operational exposure. Choose the boundary according to the action's urgency, reversibility, and duplication risk.
6. Reconcile uncertain outcomes before replaying side effects
Unknown outcomes need a separate recovery path because they concern what already happened, not merely what failed.
- Locate evidence: Use an existing downstream result reference, supported operation-status lookup, or reliable correlation record.
- Determine scope: Establish whether the entire action completed, partially completed, or did not execute.
- Choose recovery: Record completion, resume only the unfinished step, or authorise a safe repeat.
- Preserve the decision: Store the evidence and decision so another worker does not repeat the investigation or action.
A local event identifier is useful for tracking, but it does not by itself prevent duplicate downstream actions. Duplicate protection depends on the actual operation and the guarantees of the systems involved. Verify its scope and retention before relying on it.
If there is no reliable way to distinguish “not executed” from “executed but unconfirmed,” the correct automated response may be to stop. Manual review is slower, but blind replay can create a second business action that is harder to reverse.
7. Make stopped work recoverable without making replay casual
A stopped-event queue is useful only if it supports a specific decision. Group work by reason and owner: invalid source data, missing context, exhausted transient failures, unsupported types, and uncertain outcomes should not be one undifferentiated backlog.
Before releasing an event, require the reviewer to answer: What changed? Which step will run? What previous side effect must not repeat? Is the event still relevant? What evidence makes this safe?
For a widespread mapping defect, repair the rule and test a small, explicitly selected replay set before releasing the rest. For malformed input, obtain a correction rather than editing away the evidence of the original failure. For uncertain outcomes, reconcile first.
Use counts and oldest-event age together. A small blocked queue can still contain an important event that has exceeded its review deadline. Conversely, a large count of intentionally ignored event types may be expected rather than urgent.
8. Test the classification on a bounded sample
As a next action, take a recent, bounded sample of failed events and fill in the same fields used in the worked example. For each event, require a category, observable evidence, next action, owner, and stop or review condition.
Measure the share of sampled events with a complete decision, the share retried without any changed prerequisite, and the number of unknown outcomes that were replayed before reconciliation. Track overdue blocked events and exhausted retries separately.
Then test the routing against representative failure cases before enabling new automatic behaviour. A malformed event should stop; missing context should hold; an eligible transient failure should receive a bounded schedule; an uncertain side effect should reach reconciliation. Success means those decisions are visible and enforceable—not that every event eventually executes.
Put the next conversation to work.
Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.
Explore Communications