Back to all articlesWebhook processing

Record Webhook Processing States Without Hiding Unfinished Work

Alactic Editorial10 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

A webhook event can be stored without being applied, claimed without work starting, or partially applied while another step remains unresolved. A single processed flag hides these differences and makes recovery depend on guesswork.

Use an explicit event state, an effect ledger, and a time-limited claim. Completion requires durable evidence for every required effect. Recovery must also protect against two quieter failures: overwriting a renewed lease and repeating a local change that committed before its success record was saved.

This recommended design covers received webhook events, not CRM pipeline stages or a general automation framework. The example and identifiers are fictional, not Alactic customer results or evidence of product capabilities.

1. Define completion before naming states

Start with one event type and list its required effects. Otherwise, completed can mean whatever the last worker finished.

Fictional worked example: a contact-change event requires a local contact update and a durable notification request. A separate sender handles that request later. Completion means both local effects are committed; it does not mean a message was sent, delivered, or read.

The dashboard should therefore say notification request recorded, not customer notified. If delivery is the business requirement, define a different completion contract or link a downstream delivery lifecycle.

Save the contract version on each event so a software change cannot silently redefine an older completion claim. Include:

  • Required effects: the changes or handoffs that must exist.
  • Evidence: the durable records proving each effect.
  • Boundary: outcomes owned by another process.
  • Skip rule: when no work should occur and who may decide.

For the hypothetical contract C1, contact revision evidence and a committed notification request satisfy the boundary. A reserved request key does not.

2. Give each state one operational meaning

  • stored: durably recorded and eligible for a claim; no application work implied.
  • claimed: a worker holds a time-limited lease; application has not durably started.
  • applying: effects have started; the ledger shows which are complete.
  • retry_wait: unfinished work may be retried after a recorded time.
  • blocked: investigation, reconciliation, or repair is required.
  • completed: the completion contract is satisfied with evidence.
  • skipped: an explicit rule or authorised decision says not to apply the event.

A blocked event needs a reason and owner. An applying event needs effect-level evidence. Without those, the new labels merely rename uncertainty.

Keep duplicate receipts separate from processing state. Attach a repeat receipt with the same verified event identity to the existing event rather than creating independent execution. A duplicate receipt must not reset a completed event.

Completed and skipped are terminal here. Correct a mistaken terminal decision through a linked correction or scoped replay record, preserving the original history.

3. Use a filled transition contract

Reusable, filled hypothetical artifact: this table describes the contact-change handler. Replace its contract, rules, and roles for another handler; reject transitions not explicitly permitted.

From → To               Owner               Guard and required evidence
Receipt → stored        Receiver            Verified identity; durable payload reference;
                                            contract C1; receipt timestamp
stored → claimed        Claimant            Atomic eligible-state comparison;
                                            new generation G17; expiry; attempt A1
claimed → applying      Current holder      Matching unexpired generation;
                                            first effect intent recorded
claimed → skipped       Policy evaluator    Current ownership; rule R4; no effects;
                                            evaluator and reason recorded
claimed → retry_wait    Current holder      Matching ownership; known temporary failure;
                                            next retry time recorded
claimed → blocked       Current holder      Matching ownership; repair reason and owner
applying → completed    Current holder      Matching ownership; C1 evidence check;
                                            contact R88 and request N51 committed
applying → retry_wait   Current holder      Matching ownership; ledger saved;
                                            remaining work safe to retry
applying → blocked      Current holder      Matching ownership; uncertainty or conflict;
                                            reason and investigation owner
retry_wait → claimed    Claimant            Retry due; atomic eligible-state comparison;
                                            new generation and attempt
claimed/applying        Recovery process    Atomic recovery guard below; invalidate
  → retry_wait                              ownership; no unresolved outcome;
                                            remaining work safe to retry
claimed/applying        Recovery process    Atomic recovery guard below; invalidate
  → blocked                                 ownership; reconcile before release
blocked → retry_wait    Authorised operator Repair evidence; remaining work specified;
                                            conditional state update; retry time
blocked → skipped       Authorised operator Proven safe to stop; no unresolved partial
                                            effects; conditional update; reason

Recovery guard: compare the observed state and lease token or generation, and confirm expiry using authoritative time in the same atomic operation. Invalidate old ownership before releasing work for another claim. If any comparison fails, leave the event unchanged and reread it.

Receipt is an entry action, not an eighth state. Current holder means the write validates ownership, not that a worker supplies its name in a log. The table excludes applying directly to skipped: reconcile partial effects through blocked before deciding that stopping is safe.

4. Make effect evidence survive the mutation

Keep an event record containing source identity, tenant or workspace, contract version, state, attempt, lease generation and expiry, retry time, and any blocked reason and owner. Retain an append-only transition history alongside the queryable current state.

For each effect, store a stable identity, target, status, attempt association, and evidence reference. Useful statuses are not started, intended, succeeded, and outcome unknown. An error describes a failure signal, not necessarily the resulting state.

The difficult local gap occurs when a contact mutation commits and the worker crashes before recording success. A stable key in the event ledger alone does not connect that mutation to the event or prevent repetition.

Preferred design: commit the local mutation and effect-success record in one transaction. Validate current ownership within that transaction, with locking or equivalent concurrency control so lease recovery cannot interleave between validation and commit. Either both records commit or neither does.

If that transaction boundary is unavailable: the target operation must durably bind the stable effect identity to its mutation and result. For example, a target-side operation record can associate contact-update:E204 with revision R88, atomically with the contact change. Enforce identity uniqueness within the appropriate tenant and effect scope, and reject conflicting reuse.

Recovery then looks up that identity at the target. A confirmed committed result repairs the ledger without another mutation. A generic matching contact value or recent revision is insufficient: another event could have produced it.

Hypothetical recovered evidence
Event: E204 | contract: C1 | attempt: A2
Effect: contact-update:E204
Target operation: OP204 | contact: C42
Target result: committed revision R88
Ledger status: succeeded
Evidence source: durable target operation lookup

If no atomic success record or durable target association exists, do not assume the update is repeatable. Block for reconciliation or redesign the operation. Apply the same discipline to notification-request creation, using a unique effect identity and committed request evidence.

5. Walk through the partial-application example

All histories below are fictional. They illustrate expected outcomes, not measured results.

Normal completion

E201: stored → claimed → applying → completed. Contact and notification-request evidence satisfy C1. A later sending failure belongs to the linked sender lifecycle; it does not erase the handler's committed handoff.

Crash after mutation, before ledger success

E204: stored → claimed → applying → blocked → retry_wait → claimed → applying → completed. Attempt A1 commits contact revision R88 and target operation OP204 under effect identity contact-update:E204. It crashes before saving ledger success. No notification request exists.

Recovery conditionally invalidates the expired ownership and moves E204 to blocked. A reconciliation lookup finds OP204 and R88 durably associated with this exact effect. Recovery records that success, confirms the notification effect remains unfinished, and authorises release to retry_wait.

Attempt A2 acquires a new generation, reads the repaired ledger, and creates only notification request N51. Completion references R88 and N51. The observable result is one contact mutation, one request, and evidence spanning both attempts.

With a shared transaction instead, the crash would leave either both the contact mutation and ledger success committed, or neither committed. The same recovery rule reads that durable outcome rather than guessing.

Unknown external outcome

E209: stored → claimed → applying → blocked. An external operation has no confirmed outcome. Recovery does not repeat it merely because the lease expired. Reconciliation must establish a result or a safe next action.

Intentional non-action

E215: stored → claimed → skipped. Rule R4 finds no applicable change before effects begin. The rule version and reason remain visible, separate from successful application.

6. Recover ownership with an atomic comparison

Selecting expired claims is only discovery. A worker may renew its lease, or another recovery attempt may replace ownership, before the selected row is updated. Recording the old token in history does not prevent overwriting that newer state.

  1. Read a candidate. Capture state, generation, and expiry.
  2. Attempt guarded recovery. In one atomic transition, require the observed state and generation still to match and the lease still to be expired. Clear or advance ownership, write the recovery transition, and move to blocked or safely justified retry_wait together.
  3. Handle comparison failure. Make no change. Reread and reassess; never force the old decision onto the new lease.
  4. Reconcile under protected ownership. Use blocked as a holding state for uncertain outcomes. Guard reconciliation updates against competing operator or recovery changes.
  5. Release only proven-safe work. Record evidence and remaining effects before making the event claimable again.

The successful recovery transaction invalidates the old owner's authority before another claim is possible. Every local effect transaction and state write must enforce current ownership with concurrency protection; a separate check followed by an unguarded write is insufficient.

External destinations may not enforce your lease generation. Use their actual deduplication or reconciliation mechanisms where available. Lease expiry alone neither cancels an in-flight external operation nor proves that retrying is safe.

Give retries a delay policy, budget, and escalation route. HTTP 429 indicates rate limiting; a Retry-After header may provide retry timing. Respect available signals without assuming every dependency implements limits identically.

7. Control decisions and expose unfinished work

Give receivers, workers, recovery processes, and operators distinct transition responsibilities. Inspection permission should not automatically permit skipping or replaying an event.

OWASP recommends least privilege, deny-by-default access, and permission checks on every request. Apply those principles to repair tools as well as routine processing. These are implementation recommendations, not a product audit.

Report stored backlog age, expired claims, due retries, blocked age and owner, completed events, and skip reasons separately. A low error count is not reassuring when applying events stop progressing. A rising skip count may reveal an incorrect rule rather than healthy throughput.

An event table, effect records, and guarded transactions may suffice; a separate orchestration service is not inherently necessary. The essential distinction is durable evidence of success versus absence of an error.

8. Test crash boundaries before expanding

Pilot one handler with these controlled, hypothetical tests and explicit expected outcomes:

  • Crash after storage: the event remains available for one atomic claim.
  • Crash after claiming: guarded expiry recovery invalidates ownership before releasing work.
  • Renewal races recovery: recovery using the old observation changes nothing and rereads the renewed lease.
  • New claimant races stale recovery: the generation comparison fails; the newer claim remains intact.
  • Crash after mutation before ledger success: the shared transaction leaves both or neither committed; alternatively, target identity lookup repairs the ledger without repeating the mutation. Missing proof means blocked.
  • Stale worker resumes: protected local writes are rejected; uncertain external outcomes enter reconciliation.
  • Duplicate receipt: receipt history grows without another independent execution.

For every test, inspect what is proven, what remains, and who acts next. Require no completed event with missing evidence, no blocked event without an owner, and no repeated local mutation in the crash-gap test. If those answers still depend on interpreting scattered logs, strengthen the records and guards before rollout.

Webhook processingEvent statesRecovery designIdempotencyOperations

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications