Back to all articlesWebhook processing

Decide How Long to Keep Webhook Deduplication Records

Alactic Editorial9 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

A webhook duplicate arriving after its deduplication record expires can look like a new event. Keeping records longer reduces that exposure, but increases storage, cleanup work, and recovery complexity.

Choose retention from documented redelivery behavior, measured duplicate arrival ages, permitted replay practices, and the consequences of repeating work. Put delays on one clock, add a justified margin, and define how historical recovery is recognized independently of the expiring identity store. There is no universal safe duration.

1. Define the identity and its retention clock

A deduplication record answers a narrow question: has this event identity already been accepted or processed within this scope? It does not prove that every downstream action completed.

A recommended identity combines source, account or tenant scope, and a stable event identifier. An identifier unique only within one account can suppress unrelated events if account scope is omitted. A delivery-attempt identifier can miss repeated deliveries of the same event.

Event identity also differs from business identity. Separate events may legitimately concern one order. A reconstructed event may have a new identifier but request an action already performed. Event-key retention alone cannot resolve either case.

Document these clock rules:

  • Anchor: first durable receipt or claim of the identity, not source creation time.
  • Expiry: fixed from that anchor, or refreshed by duplicate arrivals. Refreshing can extend storage indefinitely for repeatedly delivered identities.
  • State: processing, completed, failed, and uncertain outcomes need distinct treatment. Unresolved work must not become fresh work merely because a completed-record timer elapsed.
  • Enforcement: specify whether suppression ends at logical expiry or physical deletion. Cleanup lag is not a dependable safety margin.

Retention is not concurrency control. Simultaneous deliveries still need an atomic claim or uniqueness mechanism, plus recovery for abandoned claims.

2. Gather evidence on the same clock

Collect current source documentation for automatic retries, manual redelivery, and historical replay. These are separate mechanisms; an automatic retry horizon says nothing about how far back an operator can replay. Record documentation dates and qualifications. Mark undocumented behavior unknown.

Measure elapsed time from first durable receipt to later receipts of the same identity. Source event age answers a different question: an old event may genuinely be arriving locally for the first time.

  • Separate automatic retries, internal queue redelivery, operator replay, and backfill where identifiable.
  • Measure duplicate ages near proposed expiry boundaries, not just total duplicate counts.
  • Include outage recovery and delayed queue drains, not only quiet operation.
  • Record telemetry gaps and operational conditions absent from the observation period.

Do not measure only duplicates recognized by the live store. After expiry, a duplicate may be counted as new. A bounded, access-controlled identity audit can expose this blind spot without retaining full payloads. Give that audit its own storage and deletion rules.

An observed maximum is evidence, not a future upper bound. It can challenge assumptions, but cannot establish that later arrivals are impossible.

3. Calculate the supported ordinary horizon

First decide which arrivals normal processing must support without exceptional review. Then calculate:

Ordinary horizon = maximum supported path age at arrival
Retention = ordinary horizon + explicit operational margin

Each path must include delays that can occur sequentially. If routine replay starts at the maximum permitted receipt age and then waits in a queue, add that queue delay. Taking the larger duration would undercount exposure.

Separate a committed delay budget from margin. The delay budget describes operation you intend to support. Margin covers a named uncertainty or contingency, such as an additional recovery cycle. Neither should be an unexplained round-up.

Historical replay can use a separate recovery path instead of forcing indefinite retention. However, routing requires surviving evidence: an expired lookup cannot reveal whether an event is an old duplicate. A trustworthy replay envelope or durable receipt ledger must supply that distinction.

Missing identity evidence does not prove that an action never happened. Where repetition matters, durable action checks or an uncertainty path must protect effects even when event classification fails.

4. Use a filled retention worksheet

Fictional worked example: all quantities, durations, observations, and operational budgets in this worksheet are hypothetical. They illustrate a decision, not provider behavior, an Alactic capability, or customer results.

Shared assumptions

  • Volume: 1,000,000 new identities daily, steady for planning; duplicates add no keys.
  • Record size: 160 bytes before database overhead; payload storage excluded.
  • Storage factor: 2.5 times raw size for indexes and overhead. Replicas, backups, audit storage, and compaction space require separate budgets.
  • Clock: fixed expiry from first durable receipt; duplicates do not refresh it.
  • Automatic retry path: source retries can reach ingress through receipt age five days, followed by up to two days of internal queue or recovery delay: seven days total.
  • Routine replay path: an authorized replay can start through original receipt age 21 days, followed by up to two days of queue or recovery delay: 23 days total.
  • Observation: a separate 120-day audit found a maximum duplicate gap of 26 days, traced to exceptional recovery.
  • Consequence: repetition could create duplicate work items or repeat an external notification.

Filled calculation and decision record

Status: hypothetical working policy
Clock: first durable receipt; fixed expiry
Automatic path: 5 days + 2 days = 7 days
Routine replay path: 21 days + 2 days = 23 days
Supported ordinary horizon: max(7, 23) = 23 days
Operational margin: 7 days
Selected retention: 23 + 7 = 30 days

Margin rationale:
Allow one additional weekly recovery cycle after the
budgeted queue delay if an ordinary recovery is missed.
This is contingency, not permission to expand routine replay.

Observed exceptional arrival: 26 days
Treatment: incidental coverage, not the margin's basis.
It remains exceptional even though its key is retained.

Accountable role: ingestion operations owner
Review triggers: replay-age expansion, delay-budget breach,
new retry behavior, or arrivals approaching expiry

The nine days between routine replay eligibility and expiry are therefore two budgeted delay days plus seven contingency days. The four days after the observed exceptional arrival are a consequence of that calculation, not a separately justified safety allowance. If the team cannot justify a weekly recovery-cycle allowance operationally, it should revise the margin rather than preserve a convenient total.

Choice A: seven days

Approximately 7 million identities require 1.12 GB raw or 2.8 GB with the planning factor, using decimal GB. This covers the assumed automatic path but not the 23-day routine path. Reject it unless ordinary replay eligibility is shortened. Historical replay would require reconciliation, and unmarked late duplicates would still need action-level protection.

Choice B: 30 days

Approximately 30 million identities require 4.8 GB raw or 12 GB with the planning factor. This covers the calculated ordinary horizon and contingency, and incidentally includes the observed exceptional arrival. Select it conditionally: enforce replay authorization and delay budgets, retain independent replay evidence, and protect consequential actions when identity evidence is missing.

Choice C: 90 days

Approximately 90 million identities require 14.4 GB raw or 36 GB with the planning factor. This covers more historical arrivals but still leaves an expiry boundary. Prefer it only if evidence shows that reduced reconciliation work justifies the extra storage and maintenance. The same historical-replay and uncertainty controls remain necessary.

This worksheet is reusable: replace the path budgets, margin rationale, volume, record size, and overhead factor together. Changing only the selected duration hides the assumptions driving it.

5. Recognize historical replay without the expired key

A recommended recovery design uses an authorized replay job with a declared source, account scope, original-receipt range, purpose, and permitted effects. The job carries preserved original receipt timestamps and event identities from a durable receipt ledger or archive with a separately defined lifetime. Authorization and metadata integrity must be checked; an arbitrary incoming replay flag is not trusted evidence.

Routing compares that preserved receipt time with the current retention boundary. A job spanning retained and expired periods is split or evaluated per event. Source creation time alone is insufficient: it cannot establish prior local receipt. If original receipt evidence is missing, classify the event as uncertain rather than inventing an age.

  1. Preview: identify affected records and the durable outcomes available for comparison.
  2. Classify: distinguish completed actions, incomplete work, and uncertain outcomes.
  3. Authorize effects: rebuilding a projection does not automatically authorize another external notification.
  4. Preserve identity: keep original event keys or a mapping when replay envelopes use new identifiers.
  5. Record recovery: keep job decisions and action outcomes so a restarted replay does not repeat authorized effects.

An unmarked duplicate arriving through ordinary ingestion after expiry may remain indistinguishable from a genuine new first arrival. The design cannot honestly promise historical routing in that case. Before consequential effects, consult durable business-action records using an appropriate action key. If those records cannot settle whether the action occurred, use an uncertainty queue for review.

These action records need their own lifetime based on the operational repetition risk. Do not assume a downstream service provides durable idempotency or retains its protection as long as you need.

6. Address failures that longer retention cannot fix

Marking an event complete before effects finish can suppress necessary recovery. Writing its identity only after effects finish leaves a crash window for repetition. Durable processing state, a transactional outbox where appropriate, and action-level idempotency can reduce these gaps; none establishes a blanket exactly-once guarantee across independent systems.

Unresolved records need age alerts and recovery rules, not ordinary completed-record deletion. Preserve what is known about partial effects before deciding whether to retry.

Restoration can also remove protection despite a sufficient configured duration. Establish whether business effects and identity records can be recovered consistently. Otherwise, pause consequential processing while reconciling the uncertain interval.

Increasing retention does not recreate deleted keys. Recover suitable evidence or keep affected traffic on the uncertainty path. Shortening retention likewise changes exposure: inspect affected age bands before bulk deletion rather than treating it as a storage-only change.

7. Test the boundary and review the evidence

Before adoption, test the following in a controlled environment. The ages below remain part of the hypothetical worksheet, not recommended universal durations.

  • Retained duplicate: replay a completed identity at age 29 days. Confirm suppression and no repeated effect.
  • Declared historical replay: replay an identity at age 31 days through an authorized job with preserved receipt metadata. Confirm historical routing despite an empty live lookup.
  • Missing or untrusted metadata: submit a replay without verifiable receipt evidence. Confirm uncertainty handling, not classification from source age.
  • Unmarked expired duplicate: submit an ordinary-ingestion duplicate at age 31 days. Confirm that durable action checks prevent repetition, or that uncertainty blocks the consequential effect.
  • Genuine first arrival: submit an old source event with no prior receipt. Confirm source age alone does not cause rejection; evaluate its intended action separately.
  • Unresolved processing: age an unfinished claim past normal expiry. Confirm it enters recovery rather than becoming fresh work.

Monitor duplicate ages near expiry, queue-delay breaches, unresolved processing age, exceptional replay volume, deletion lag, and confirmed repeat effects. Review when retry behavior, replay eligibility, or recovery practices change.

The deliverable is a recorded calculation, an accountable recovery owner, and observable test results. A database expiry setting without those boundaries is not a complete retention policy.

Webhook processingDeduplicationData retentionOperational recovery

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications