Back to all articlesWebhook processing

Prevent Repeated Side Effects After a Webhook Worker Crashes

Alactic Editorial9 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

A worker updates a customer record, requests an external action, and crashes before recording success. On retry, the system sees unfinished work even though some effects may already have happened. Marking the event complete earlier merely exchanges the risk of duplicate actions for missing actions.

The useful recovery boundary is each individual side effect. Commit local changes with their completion records, use stable action keys where an external service documents safe replay, and hold ambiguous consequential actions for reconciliation. Crucially, a replacement must not be released while an earlier attempt could still execute.

1. Separate an unknown outcome from a failed action

For each action, identify when its effect becomes real and when your application durably records evidence of that effect. A crash between those moments creates uncertainty.

A local update can share a transaction with its completion record when both live in the same transactional database. An external request generally cannot. Your database can show unfinished work while another service has already accepted the request.

Writing a started record before sending helps investigation, but does not settle recovery. It might mean the worker never sent, sent without receiving a response, or received a response without saving it. Use three distinctions:

  • Known applied: reliable evidence establishes the effect. Recover its reference rather than repeat it.
  • Known not applied and unable to execute later: evidence establishes both non-execution and the end of any earlier attempt. A new attempt may be considered if still authorised.
  • Unknown or still executable: use documented safe replay or retain a review hold.

A timeout proves neither failure nor cancellation. An expired lease allows work ownership to change; it does not establish what happened externally or stop a paused worker from resuming.

2. Persist intent and track completion per action

Persist an accepted webhook in a durable inbox before acknowledging receipt. Arrange recovery so a crash after that commit cannot strand the work. Scanning pending inbox records is one possible design; a transactional outbox is another when database writes and queue publication need coordination.

Create separate action records for the intended effects. Keep these fields:

  • Identity: workspace, source event, business object, action type, and version.
  • Intent: authorised operation and a stable payload or fingerprint.
  • Recovery: action key, attempt history, external reference, and available lookup evidence.
  • State: pending, executing, confirmed, failed, or held for reconciliation.
  • Ownership: worker claim, lease expiry, and concurrency token where supported.

Enforce uniqueness on the intended action identity. Otherwise, duplicate webhook deliveries can create separate action records with different keys, and both can execute.

An event identifier is not always the right identity: multiple events might describe one business operation, while one event might require several effects. Define identity around the operation you intend to perform.

Keep action state separate from event completion. A hold is not success. External acceptance may also differ from the final business outcome, so record which milestone your completion state represents.

3. Worked example: a filled crash-window matrix

Fictional worked example: Event evt-482 concerns job job-731. Its authorised workflow changes a local CRM status, creates a scheduling task, and requests a physical dispatch. The services and capabilities below are hypothetical, not Alactic features or customer results.

The hypothetical scheduling service documents idempotent creation and lookup. The dispatch service offers no documented replay protection. This filled matrix is a reusable runbook artifact: replace the identifiers, evidence requirements, and service contracts for another workflow.

Row A: Local record update

  • Effect and identity: Mark the job scheduled; workspace-A/job-731/mark-scheduled/v1.
  • Crash window: Before commit, neither the mutation nor completion should persist. After commit, both should persist even if worker acknowledgement is lost.
  • Protection: One transaction containing the guarded update, transition record, and action completion.
  • Evidence: Committed action record and matching transition.
  • Recovery: Retry an uncommitted transaction; skip a confirmed action. Investigate inconsistent records rather than overwrite them.

Row B: External action with documented idempotency

  • Effect and identity: Create one scheduling task; workspace-A/job-731/create-task/v1.
  • Crash window: The service may accept the task before the local result is committed.
  • Protection: Persist the request key before sending; reuse it with the unchanged payload under the service's documented contract.
  • Evidence: Original task reference recovered through documented replay or lookup.
  • Recovery: Replay only within documented scope and retention conditions. If those conditions cannot establish safety, hold for reconciliation rather than generate a fresh key.

Row C: External action requiring reconciliation

  • Effect and identity: Request one dispatch; workspace-A/job-731/request-dispatch/v1. This local identity provides no external duplicate protection.
  • Crash window: Dispatch may already exist, or an earlier worker or request may still submit it.
  • Protection: Save authorised intent before sending; hold unresolved attempts instead of replaying automatically.
  • Evidence: A matching dispatch reference, or authoritative evidence that no dispatch occurred and that the earlier attempt cannot execute later.
  • Recovery: Record a confirmed dispatch without repeating it. Release a replacement only after resolving every earlier attempt and rechecking authorisation. A negative search, expired lease, or revoked local claim alone is insufficient. If terminal resolution or prevention cannot be established, retain the hold and escalate.

All three rows have action identities. Only Row B assumes that the external service uses that identity to protect against repeated execution.

4. Make local effects and completion one commit

For Row A, place the business mutation and completion write inside one database transaction. Enforce action uniqueness and use locking or conditional updates so competing workers cannot both apply the transition.

begin transaction
  lock the unique action record
  if already confirmed:
    finish without repeating the mutation
  otherwise:
    verify the allowed business transition
    update the business record
    record the transition
    mark the action confirmed
commit

This is a conceptual sequence, not implementation-ready code. Its locking and uniqueness rules must match the database's actual behaviour.

Do not call an external service inside the transaction and assume rollback will undo the call. Local rollback cannot reverse an independent service's accepted request. Long network waits also extend lock duration without removing uncertainty.

If the local change requires later external work, persist that intent in the same transaction. This prevents the follow-up from being lost, but does not make its eventual external execution happen only once.

5. Verify the external replay contract

For Row B, check the service's documented behaviour before depending on its key. Verify supported operations, key scope, retention period, concurrent-request handling, changed-payload handling, and responses for failed or in-progress requests. Also establish how to recover the original reference.

Persist the key before the first request. Generating a new key on retry creates a new operation rather than recovering the original one. Preserve the business payload too; a changed intention needs an explicit decision about a new action version.

A worker lease reduces overlap but is not external duplicate protection. A paused worker can resume after lease expiry. A local concurrency token can reject its stale database write without preventing its external call. Preventing that call requires a control enforced at the relevant execution boundary, not merely a local ownership update.

Use bounded retries and preserve unresolved state when the retry budget ends. Stopping retries does not prove non-execution. If replay protection expires before recovery, move to reconciliation rather than silently switching keys.

6. Release a review hold only after earlier attempts are resolved

For Row C, assemble a review record containing the intended effect, job and action identities, all attempts, timing, references, current worker status, and uncertainty. Restrict review and release to authorised people and include only the sensitive information needed.

Give reviewers distinct resolution paths:

  1. Confirmed applied: Attach evidence and the external reference. Do not repeat the action.
  2. Confirmed not applied and no earlier attempt remains executable: Document evidence for both conclusions. Recheck that the dispatch is still wanted and authorised, then release one controlled replacement.
  3. Still uncertain or still executable: Retain the hold, assign an owner, and escalate with a specific investigation step.
  4. No longer wanted: Close pending intent with a reason, while separately resolving any earlier in-flight attempt or discovered effect. Closing locally does not cancel external work.

The second path requires more than a search returning no dispatch. A delayed request may arrive after the search, or a paused worker may resume after review. A revoked claim or expired lease does not establish that either path has stopped.

Acceptable evidence depends on the execution system. It might be an authoritative terminal rejection, or a confirmed cancellation or execution barrier that covers the earlier request and prevents later acceptance. A cancellation request by itself is not confirmation. Record what the evidence covers, including every unresolved earlier attempt.

If the external service cannot establish a terminal outcome or prevent the old attempt from executing, keep the hold and escalate. Do not translate “not found yet” into “confirmed not applied.” Another lookup may help investigation, but does not fix an unresolved execution path.

Release must also be serialised: two reviewers must not independently authorise replacements. Record the evidence, reviewer, decision, and replacement attempt together under the application's concurrency controls. Cancellation or compensation is another consequential action, not a harmless recovery shortcut.

7. Test delayed execution during review

Duplicate webhook delivery alone does not test these boundaries. In a controlled environment, interrupt processing after durable receipt, around the local commit, before external sending, and after external acceptance but before saving the result. Also test overlapping claims and expired replay protection.

Hypothetical controlled test for Row C:

  1. Pause the original worker immediately before its external submission. Separately test a request delayed after leaving the worker.
  2. Expire the lease and route the unresolved action to review.
  3. Return no dispatch from the test lookup. Attempt review release while the earlier worker or request remains capable of execution.
  4. Verify that release remains blocked despite the negative lookup and expired claim.
  5. Resume the delayed path. If it executes, reconcile that original dispatch without creating a replacement. If it becomes authoritatively unable to execute and no dispatch occurred, record that evidence before permitting an authorised replacement.

The expected outcome is a hold while the earlier attempt remains executable, not merely a clean local status. If the test environment cannot simulate the external boundary, record that coverage gap rather than claim the risk has been tested.

Also interrupt after review resolution but before updating the event summary. Recovery should preserve the decision without releasing another attempt.

8. Use evidence, not queue clearance, to judge recovery

Start with one consequential workflow and assign every side effect a recovery rule. Include secondary effects such as notifications, audit entries, and follow-up tasks. For each, record its identity, effect boundary, evidence, protection mechanism, and authorised recovery path.

Observe unresolved-action age, hold backlog, attempts without external references, and discovered repeats. Keep enough history to distinguish a new business request from recovery of an old one. A falling queue count is not evidence that effects were handled safely.

Run a controlled interruption test for each action. Any path that moves from unknown to replacement without documented replay safety or resolution of earlier execution needs a revised boundary before unattended retries are enabled.

Webhook processingIdempotencyRecovery workflowsOperational reliability

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications