Back to all articlesWebhook processing

Separate Source Redelivery From Internal Webhook Retries

Alactic Editorial9 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

A webhook arrives, your endpoint saves it, and a worker begins processing. The source does not receive your acknowledgement, so it sends the event again. Meanwhile, the worker encounters a temporary failure and schedules its own retry. If both paths create fresh work, one business event now has competing recovery loops.

The practical answer to webhook redelivery versus internal retry is to put the ownership handoff at durable acceptance. Before that boundary, your system has not taken responsibility for processing. After it, your internal recovery system owns progress, even if the source continues redelivering. Later deliveries should resolve to the existing event record, not start another independent workflow.

This is a recommended architecture, not a claim about a particular product control. It depends on durable records, atomic state changes, and an explicit recovery path for unfinished work.

1. Define the acceptance boundary before choosing retry rules

Durable acceptance means more than receiving an HTTP request or placing data in process memory. It means committing enough information to recover the event after the receiver crashes: its identity, necessary payload or durable reference, processing state, and a reliable indication that work remains.

A useful ownership rule is:

  • Before acceptance: the source remains the transport recovery path where its documented redelivery behavior supports that. Your receiver must not claim acceptance when the commit failed.
  • After acceptance: your internal scheduler, queue, or recovery scanner owns unfinished processing. Source redelivery becomes another observation of the same event.
  • After completion: neither path should repeat the completed operation. An intentional replay needs a separate, controlled decision.

Your system cannot guarantee that a source will retry. Check its acknowledgement contract and provide reconciliation for missing events where necessary. Equally, you cannot assume the source will stop immediately after acceptance: the response can be lost, or another delivery may already be in flight.

Separate acceptance from business success in your endpoint contract. If the source supports asynchronous acceptance, return its documented successful acknowledgement after the acceptance commit, rather than waiting for the worker. If it requires a different contract, document that constraint instead of assuming a generic success response is sufficient.

2. Track three identities, not one attempt counter

A single retry count cannot explain whether the source repeated a request or your worker repeated an operation. Keep separate identities and link them.

  • Event key: the stable identity used to recognise the same source event. Scope it to the source and tenant or account as appropriate; an identifier from one account must not collide with another.
  • Delivery receipt: your record of each incoming request, including any source-provided delivery identifier, arrival time, acceptance decision, and response outcome when observable.
  • Processing attempt: a worker execution against the canonical event record, with its attempt identifier, claim or lease, result, and next recovery state.

Verify the source's identifier semantics. A delivery identifier may change on redelivery and therefore may not be an event deduplication key. Conversely, a business object identifier may cover multiple legitimate changes and be too broad. Do not silently substitute a payload hash unless the event contract makes that safe; equivalent events can have different serialization, and identical-looking payloads can represent distinct actions.

Also distinguish event identity from operation identity. Two different events can request the same business operation. If that is possible, give the operation its own uniqueness guard rather than expecting event deduplication to solve it.

3. Filled fictional retry-ownership map

Fictional worked example: every identifier, timestamp, count, and business detail below is hypothetical. A source emits an event asking a business system to mark an existing CRM task as ready for review. The recommended design keeps the task update and completion record in the same database transaction.

The receiver commits the event, but its first acknowledgement is lost. A worker then fails before committing the task update. A source redelivery overlaps with the internally scheduled retry.

Canonical event key: source-north / account-demo / evt-742
Operation key: task-286 / mark-ready / revision-6
Intended result: task-286 has status ready-for-review

Time      Observation                 Durable decision                 Recovery owner
09:00:00  Receipt R-01 arrives         Event E-742 accepted; pending    Internal system
09:00:01  Success response sent        Source receipt is unknown        Internal system
09:00:02  Worker attempt A-01 starts  E-742 claimed with lease L-01    Internal worker
09:00:03  A-01 fails before commit    E-742 retry_wait; retry due       Internal scheduler
09:00:04  Receipt R-02 arrives         Matches E-742; no new work       Internal scheduler
09:00:05  R-02 acknowledged           Existing acceptance confirmed    Internal scheduler
09:00:08  Worker attempt A-02 starts  E-742 claimed with lease L-02    Internal worker
09:00:09  A-02 commits                Task updated; C-742 inserted     No routine retry
09:00:12  Receipt R-03 arrives         E-742 completed; no new work     None

Single completion record:
  completion_id: C-742
  event_key: source-north / account-demo / evt-742
  operation_key: task-286 / mark-ready / revision-6
  completed_by_attempt: A-02
  result: task status committed as ready-for-review

Linked evidence:
  delivery receipts: R-01, R-02, R-03
  processing attempts: A-01, A-02
  business completion records: C-742 only

The map shows three deliveries but only two processing attempts and one completion. Those counts are not a performance target; they make ownership visible. R-02 does not reset the internal retry schedule. R-03 does not clear completion or create a new operation.

The receiver can record that it sent an acknowledgement, but not infer that the source received it. That distinction explains why a correctly accepted event can still be redelivered.

4. Make acceptance and scheduling recoverable together

The dangerous gap is between saving the event and scheduling its processing. If the receiver saves the event, acknowledges it, and then crashes before publishing to a queue, the source may stop while no worker knows work exists.

Choose an acceptance design that closes that gap:

  • Database-backed pending work: commit the event as pending, then let workers or a recovery scanner discover pending records.
  • Transactional outbox: commit the event and an outbox entry together. A separate publisher sends the work notification and retries publication as necessary.

Both designs still need duplicate-safe processing. An outbox publisher can send a notification more than once if it crashes after publishing but before recording publication. The queue message should point to the canonical event, not act as permission to repeat the business action.

Enforce event uniqueness at the storage boundary. A read-then-insert check alone can race when simultaneous deliveries both see no record. Use an atomic uniqueness mechanism so only one accepted record wins; the other request should resolve to it.

If a repeated event key arrives with incompatible content, do not overwrite the accepted payload or casually label it a harmless duplicate. Preserve the conflict, prevent unreviewed execution, and route it for investigation under the source's event contract.

5. Let worker state, not redelivery, control recovery

Use a small state model with clear transitions: pending, running, retry-wait, completed, and needs-review are often sufficient. Queue messages are notifications to inspect that state, not an independent source of truth.

  1. A worker loads the canonical event and exits without executing if it is completed.
  2. It acquires an atomic claim or bounded lease. Another worker with a valid competing claim must not start the same operation.
  3. It verifies that the operation has not already completed and performs the permitted work.
  4. It records the outcome and either completion, a scheduled retry, or a review state.

A redelivery handler should not convert retry-wait back to pending merely because another request arrived. That bypasses the worker's recovery decision and lets the source cadence dictate internal execution.

Leases also need a stale-worker rule. A worker can pause beyond its lease and resume after a replacement starts. Where possible, require the current claim token when committing changes, so an expired worker cannot overwrite a newer result. This protects database transitions; it does not automatically protect an external service call.

When automated attempts are exhausted, internal ownership should become explicit review ownership, not disappear. Keep the accepted event, failure reason, last attempt, and authorised recovery action visible. Source redelivery is not a substitute for that queue.

6. Treat uncertain side effects differently from known failures

The fictional example can commit its local task update and completion together. External side effects are harder. A remote action may succeed even though the worker times out before receiving the response. Recording failure and blindly retrying can repeat that action.

Use a stable operation key for an external idempotency facility where one is documented and suitable. Verify its scope, retention, and response semantics rather than assuming indefinite protection. Keep the key stable across internal attempts and source redeliveries.

Where possible, reconcile the remote outcome using a stored operation reference before trying again. If there is no safe deduplication or reconciliation mechanism, move an uncertain outcome to review instead of treating it as a confirmed failure.

This is the trade-off: stopping for review can delay completion, but automatic repetition can create a second business action. A single local completion record alone does not prove that only one external action occurred.

For workflows with several side effects, track each step's outcome. If one step completed and another failed, resume the unfinished step rather than restarting the whole workflow. A request to replay completed work should name the intended operation, require appropriate authorisation, and retain a link to the original event.

7. Test the ownership boundary with deliberate failures

Test recovery at the handoff points, not only by resending a successful request. The following are recommended checks, with expected observations rather than promises of failure-free execution.

  • Commit succeeds, acknowledgement is lost: redelivery links to the existing event; it does not create another operation.
  • Acceptance storage fails: the receiver does not report durable acceptance; monitoring exposes the failure and any source recovery remains governed by its contract.
  • Publishing fails after acceptance: pending-work discovery or the outbox eventually exposes the event to a worker without relying on source redelivery.
  • Worker crashes before completion: claim recovery makes unfinished work eligible again, while completed steps remain protected.
  • Two workers overlap: claims and commit guards prevent both from finalising the same local operation.
  • External response is lost: the event enters reconciliation or review rather than an unconditional retry.
  • A completed event is redelivered: the receipt is recorded, the acknowledgement follows the source contract, and no new processing attempt is needed.

Also test retention. Deleting deduplication records while old events can still arrive can turn a historical redelivery into apparently new work. Choose retention using the source's documented replay behavior and your own replay procedures. Where practical, retain a minimal completion marker longer than detailed payloads, subject to your data-handling requirements.

8. Use the map to make one measurable change

Start with one webhook event type that has shown overlapping recovery. Fill the map with actual receipt and worker records, leaving unknowns visibly unknown. For each accepted event, identify the current recovery owner and evidence of completion or an outstanding recovery action.

Then measure separate quantities:

  • Incoming delivery receipts per canonical event.
  • Processing attempts per canonical event.
  • Accepted events with neither completion nor a scheduled, active, or review-owned recovery action.
  • Duplicate receipts that incorrectly create fresh processing work.
  • Uncertain external outcomes awaiting reconciliation.

Your next change should close one demonstrated gap: add an atomic event-key constraint, connect acceptance to recoverable scheduling, or stop duplicate receipts from resetting worker state. Re-run the corresponding failure test and inspect the linked records. The useful outcome is not fewer HTTP requests; it is that every accepted event has a clear owner, and completed work stays completed when another delivery arrives.

WebhooksRetry ownershipIdempotencyEvent processing

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications