Build a Review Queue for Webhook Events That Cannot Progress
Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.
A dead-letter queue can tell you which webhook events stopped processing without telling you what anyone should do next. A failure message such as record not found leaves important questions unanswered: Which record was expected? Did an earlier attempt change anything? Who can approve another attempt?
Build a separate review process around those questions. Each blocked event needs an evidence packet, one accountable resolution owner, and an explicit disposition: correct and retry, approve a skip, or investigate before taking further action. Keep the event available for diagnosis while recording decisions outside the raw queue entry.
The practical goal is not an empty queue at any cost. It is a queue in which every unresolved event has a known reason for waiting, an owner, and a next check. The worksheet below shows how to make that concrete.
1. Separate transport failure from a review decision
Moving an event into a dead-letter queue is a processing outcome, not a business decision. It does not establish that the event is invalid, that its intended change is unnecessary, or that replaying it is safe.
A useful webhook dead letter queue review process separates three questions:
- What happened technically? Record the failing processing stage, error evidence, and attempt history.
- What remains unresolved? Identify the intended business effect and whether any part of it already occurred.
- What action is authorised now? Name the person responsible for proposing a disposition and the person permitted to approve it.
Preserve distinctions between waiting for evidence, waiting for a correction, waiting for approval, and waiting for verification. A single label such as pending conceals these differences and makes queue age difficult to interpret.
Keep the original event immutable where practical. Corrections should produce a documented replay input, mapping revision, or processing change rather than silently replacing the evidence that explains the failure.
2. Require a review packet, not just an error string
A reviewer should be able to understand the blocked action without searching unrelated logs or asking the event's original handler to reconstruct it from memory. Define a minimum packet before assigning a disposition.
- Event identity: Source, event identifier, event type, receipt time, workspace or tenant, and correlation reference. If no stable source identifier exists, document how this event is distinguished from similar deliveries.
- Failure evidence: Processing stage, error classification, sanitised response or log reference, and the time of the most recent failure.
- Prior attempts: Attempt times, relevant processor or configuration version, results, and any operator intervention.
- Affected records: Intended target, current record state, dependencies, and any other records that could be affected by replay.
- Side-effect evidence: What is known to have happened, what is known not to have happened, and what remains uncertain.
- Proposed disposition: Recommended action, prerequisites, expected outcome, and stopping conditions.
- Ownership and rationale: Resolution owner, approver when required, supporting reasoning, and next review point.
Store references to restricted payloads and logs rather than copying sensitive content into a broadly visible worksheet. Include enough context to review the decision, not every field in the original request.
A missing field should be explicit. Write “side-effect status unknown; downstream lookup required,” rather than leaving the side-effect section blank. Unknown information is itself a reason to withhold replay.
3. Assign ownership without granting blanket replay authority
Give each packet one resolution owner. That person is accountable for collecting evidence and moving the decision forward; they do not need permission to perform every possible corrective action.
Separate responsibilities where the consequences warrant it:
- Queue coordinator: Checks packet completeness and routes the event to the right owner.
- Resolution owner: Investigates the failure and proposes a disposition.
- Domain reviewer: Confirms whether the intended business outcome is still correct.
- Authorised executor: Carries out the approved correction, replay, or skip.
A small team may combine roles for low-risk work. The trade-off is reduced independent review. For actions that could send communications, overwrite important records, or cause duplicate downstream work, require a distinct approval step appropriate to the risk.
Queue access must not become a shortcut around application permissions. OWASP recommends least privilege, deny-by-default behaviour, and permission checks on every request in its authorization guidance. Apply that principle to review actions as well as normal processing: permission to inspect an event should not automatically grant permission to replay it.
When an owner hands off a packet, record the receiving owner and the next action. Adding a comment that someone else should look at it is not a completed handoff.
4. Choose among correction, approved skip, and investigation
Correct and retry
Choose this path when the failure has a supported explanation, its prerequisite has been corrected, and replay remains appropriate. The packet should show how duplicate effects will be prevented or detected and how success will be verified.
Do not treat another attempt as a diagnostic substitute for missing evidence. If an earlier request timed out after reaching a downstream system, first establish whether that system completed the action. A timeout alone does not tell the reviewer whether the intended effect occurred.
For rate-limited requests, HTTP 429 indicates rate limiting, and a Retry-After header may indicate when to retry. Providers differ in how they apply limits, as described in MDN's HTTP 429 reference. Capture that evidence instead of repeatedly replaying immediately.
Approve a skip
A skip is a deliberate decision not to apply this event. Require evidence that its intended effect is unnecessary, superseded, invalid, or already represented through a verified equivalent outcome. Record who accepts that disposition and why.
Skipping should not erase the event or imply successful processing. Preserve a terminal state such as “closed—approved skip,” with the evidence and approval attached. This allows later reconciliation to distinguish intentional omission from unexplained loss.
Investigate
Choose investigation when identity, authority, ordering, partial completion, or business meaning remains uncertain. Give it a specific question and an owner: “Confirm whether the downstream record was created” is actionable; “needs investigation” is not.
Investigation is not a permanent parking state. Set a next review point and name the evidence needed to choose another disposition. If that evidence cannot be recovered, escalate the uncertainty to an accountable decision-maker rather than silently assuming a safe outcome.
5. Filled hypothetical exception-review worksheet
Fictional worked example: All identifiers, times, counts, record details, and outcomes in this worksheet are hypothetical. It describes a recommended operating design, not an existing product control.
A webhook should associate an incoming service request with a CRM organisation. Processing stopped because the incoming organisation reference had no local mapping. The proposed resolution is a narrow mapping correction followed by a reviewed replay.
REVIEW PACKET: EXC-042 Status: Awaiting approval for corrected replay EVENT IDENTITY Source: Fictional service-request system Source event ID: evt_demo_781 Event type: service_request.created Received: 08 June, 09:12 UTC (hypothetical) Workspace: demo-workspace-north Correlation reference: corr_demo_192 Original payload: Restricted evidence reference PAY-042 FAILURE EVIDENCE Stage: Resolve source organisation to CRM organisation Error: organisation_mapping_missing Sanitised detail: source organisation org_demo_19 has no mapping Evidence references: LOG-042-A and mapping snapshot MAP-042-A Last failure: 08 June, 09:22 UTC (hypothetical) PRIOR ATTEMPTS 09:12 — Mapping lookup failed; no target write reached 09:14 — Same failure; unchanged mapping configuration 09:22 — Same failure; moved to exception review Operator replay: None Processor revision: demo-r17 AFFECTED RECORDS Intended service request: req_demo_553 Source organisation: org_demo_19 Proposed CRM organisation: crm_demo_204 Current target service-request record: Not present in checked scope Dependency: Source-to-CRM organisation mapping Related waiting event: evt_demo_782, a later update for req_demo_553 SIDE-EFFECT CHECK Attempt traces stop before the target-write stage. Target lookup found no service request for this source reference. No outbound communication was part of this event's workflow. No duplicate found in the reviewed scope. Evidence references: TRACE-042 and LOOKUP-042 PROPOSED DISPOSITION Correct and retry, subject to approval. Add the verified mapping org_demo_19 → crm_demo_204. Preserve the original event identity and duplicate-detection key. Keep evt_demo_782 waiting until creation is verified. Do not batch-replay other events sharing this error label. OWNERSHIP Resolution owner: Integration operations lead Business reviewer: CRM data steward Executor: Authorised integration operator Next check: After mapping approval; otherwise next staffed review REVIEWER RATIONALE The data steward confirmed the organisation match using the restricted source record and CRM record, not name similarity alone. Available traces and target lookup support no completed write. The intended service request remains required. Replay is limited to this event because other missing mappings may have different causes. APPROVAL Pending. No correction or replay is authorised by this worksheet alone. VERIFICATION AND STOP CONDITIONS After approved replay, confirm one service-request record exists, references crm_demo_204, and retains the source request reference. Record the resulting target ID and processing result. Stop if the mapping is disputed, a duplicate is found, or any unexpected downstream action is observed. Review evt_demo_782 separately after creation is confirmed.
The worksheet makes the uncertainty visible before execution. It also prevents a common shortcut: treating every event with the same error text as having the same remedy. A shared failure signature is useful for grouping work, but it is not proof that every affected record should receive the same correction.
6. Make the disposition executable and recoverable
An approved decision should specify its scope. Identify the event, the correction, the permitted action, the person executing it, and the expected observable result. If the evidence or affected records change before execution, return the packet for review.
Use a controlled sequence:
- Recheck current target state and relevant dependencies.
- Apply only the approved correction.
- Record the replay attempt against the original review packet.
- Verify the intended business state, not merely that the worker returned without error.
- Release dependent events only after their prerequisites are confirmed.
Keep execution and verification separate in the queue state. “Replay submitted” is not equivalent to “resolved.” If the replay fails differently, append the new evidence and reopen the decision rather than overwriting the earlier diagnosis.
If a replay reveals an unexpected effect, stop further related replays and preserve the evidence. Do not automatically reverse the change: reversal can have its own consequences and should follow a separately authorised corrective plan.
Batch review can reduce repeated evidence gathering, but batch execution increases the scope of a mistaken assumption. Group packets by a proven shared cause, then retain record-level checks for permissions, current state, duplicates, and dependencies.
7. Close with evidence, then measure the waiting
Define closure evidence for each disposition. A successful retry needs the resulting state and a verification reference. An approved skip needs the approver, rationale, and any follow-up reconciliation. Investigation remains open until it produces a supported decision or an explicitly accepted unresolved outcome.
Measure whether the review process makes work decidable, not just whether it reduces queue depth:
- Packet completeness: Share of open packets with identity, evidence, owner, and next action.
- Unowned age: Time events spend without an accountable resolution owner.
- Decision waiting: Time spent awaiting evidence, correction, approval, or verification, measured separately.
- Reopened resolutions: Cases reopened because the stated outcome was not achieved.
- Recurrence: New exceptions attributable to a previously understood cause.
A rising approved-skip count deserves review even when queue depth falls. It may represent valid superseded work, or it may conceal pressure to clear the backlog. Sample the rationale and evidence rather than treating the count as success on its own.
8. Start with a bounded review pilot
For the next staffed review session, select a manageable set of blocked events across different failure types. Create a packet for each, assign a resolution owner, and require a next action before the session ends. Do not make replay volume the session's success criterion.
Hypothetical pilot target: Review ten events and leave every one with an owner, an explicit evidence gap or proposed disposition, and a scheduled next check. Record how many were initially missing side-effect evidence and how many required a domain reviewer.
Use those observations to improve the packet. If reviewers repeatedly need the same missing trace, add that evidence to packet preparation. If approvals stall because authority is unclear, define the decision boundary before expanding the queue. The immediate outcome should be visible: fewer unexplained waiting states and no closed event without a recorded reason.
Put the next conversation to work.
Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.
Explore Communications