Set the Right Boundary for Webhook Deduplication Keys
Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.
A webhook receipt marked duplicate can hide a real loss: an event from one account was accepted, and a valid event from another account was suppressed because both carried the same identifier. Test traffic sharing production storage and overlapping subscriptions can create similar mistakes.
Establish the source’s documented identifier scope rather than assuming uniqueness across accounts, environments, or subscriptions. Some identifiers may be globally unique by contract; others require additional boundaries. The right key preserves both outcomes: independent events remain distinct, while repeated deliveries of the same event match.
This guide assumes an event identifier has already been selected. It addresses namespace boundaries for receipts, not identifier selection or general workspace access controls. The workflow is a recommended design, not a claim about controls available in a particular product.
1. Define the operation your duplicate check protects
Write one sentence describing the protected operation. Accepting each source event once into a shared pipeline differs from running each subscription-specific workflow once. Both can be reasonable, but one uniqueness constraint should not silently stand in for both.
- Event identity: which source event a receipt represents.
- Delivery identity: which attempt or route brought it to the receiver.
- Processing identity: which downstream action must be performed.
A retry normally matches the earlier event receipt. Another subscription delivering that event may also need to match. Nevertheless, an independent workflow can still require its own processing record.
Adding every subscription ID to the receipt key can turn one event into several accepted events. Omitting subscription-specific processing records can cause the opposite failure: acceptance through one route suppresses work required by another. A shared event record with separately identified actions often makes these obligations easier to inspect.
2. Evaluate each namespace dimension
Source: distinguish producers when their identifiers can overlap in shared storage. Use a stable namespace rather than a display name or routing hostname. If a documented global identifier contract makes this unnecessary for uniqueness, retain source information as useful provenance.
Account: include the source account when identifier scope requires it. Use a stable account identity, not an editable label. A local workspace is not automatically equivalent: it may connect several source accounts, while one source account may feed several destinations.
Environment: preserve independent test and production namespaces where applicable. Specify whether this means producer mode, receiver deployment, or a mapped integration environment. If both producer mode and receiver deployment matter, represent them separately. Separate databases can supply isolation, but document that dependency before storage is consolidated.
Subscription: determine whether subscriptions carry copies of the same event or define independent identifier namespaces. For overlapping delivery routes, omit the physical subscription ID from event identity and retain it on delivery records. For subscription-local identifiers, include the subscription boundary. Independent workflow obligations belong in processing identity unless separate event acceptance is intentional.
Event type: include it when identifiers are unique only within a type. Otherwise, adding type can weaken matching if equivalent deliveries use different labels. Do not substitute a downstream action name for the source event type.
A logical receipt domain can distinguish independently operated pipelines that each require their own acceptance record. Keep it stable across endpoint or subscription replacement; creating a new domain for every route defeats shared deduplication.
3. Complete a key-scoping worksheet
Fictional worked example: all names, identifiers, fixtures, and expected outcomes in this article are hypothetical, not customer results. LedgerBridge sends events from North and South accounts into shared storage. Its assumed contract makes event IDs unique within each source account and producer environment, across types and subscriptions.
Two subscriptions overlap: sub-orders receives order events, while sub-all includes those same events. Both feed one pipeline. The following filled artifact can be copied and replaced with evidence from a real integration.
FILLED HYPOTHETICAL KEY-SCOPING WORKSHEET Protected operation: Accept each source event once into the commerce pipeline. Selected identifier: Verified source event_id. Source: ledgerbridge INCLUDE: other producers share the receipt store. Source account: acct-north or acct-south INCLUDE: the assumed contract permits cross-account reuse. Producer environment: test or production INCLUDE: the assumed contract permits cross-environment reuse. Receiver deployment: one shared receiver in this example. Physical subscriptions: sub-orders and sub-all OMIT from event identity: these are overlapping routes. Retain both on delivery records. Logical receipt domain: commerce-ingest INCLUDE: stable acceptance boundary for this pipeline. Both subscriptions map here. Event type: order.created, order.updated, and others OMIT from identity: IDs span types under this contract. Retain the original type as metadata. Canonical identity tuple: [source, account, environment, receipt_domain, event_id] Filled tuple: ["ledgerbridge", "acct-north", "production", "commerce-ingest", "evt-1042"] Representation metadata, outside identity: key_format = structured-tuple-v2 Required evidence before implementation: Identifier contract; authenticated account mapping; environment mapping; subscription overlap map. Review owner: integration lead Review trigger: contract, routing, or namespace change.
The format label describes how the identity is represented; it does not make the event a different event. The next sections test both that decision and the conditional subscription and type rules.
4. Test matching and separation with concrete pairs
For the main example, compare these hypothetical receipts against the filled tuple:
- Different account: South production’s
evt-1042produces[ledgerbridge, acct-south, production, commerce-ingest, evt-1042]. It differs from North’s tuple and must be accepted independently. - Different environment: North test produces
[ledgerbridge, acct-north, test, commerce-ingest, evt-1042]. It must not suppress production. - Overlapping subscriptions: North production’s event through either subscription produces
[ledgerbridge, acct-north, production, commerce-ingest, evt-1042]. Both deliveries match one event record. - Retry: another delivery through
sub-ordersproduces that same tuple. Its attempt timestamp changes delivery metadata, not identity. - Different event:
[ledgerbridge, acct-north, production, commerce-ingest, evt-1043]differs and must be accepted.
Now test the opposite subscription decision and the event-type branch. These hypothetical fixtures assume different source contracts; they are not exceptions to the main example’s contract.
ALTERNATIVE CONTRACT: subscription-local IDs
Tuple order:
[source, account, environment, domain, subscription, event_id]
A: [routefeed, acct-north, production, commerce-ingest,
sub-east, evt-1042]
B: [routefeed, acct-north, production, commerce-ingest,
sub-west, evt-1042]
Expected: DIFFERENT. Both events are accepted.
Reason: each subscription owns an independent ID namespace.
A retry of A still matches A.
ALTERNATIVE CONTRACT: type-local IDs
Tuple order:
[source, account, environment, domain, event_type, event_id]
A: [typefeed, acct-north, production, commerce-ingest,
order.created, evt-1042]
B: [typefeed, acct-north, production, commerce-ingest,
refund.created, evt-1042]
Expected: DIFFERENT. Both events are accepted.
Reason: IDs are unique only within each event type.
A retry of A still matches A.These pairs make omission conditional and testable. A key with more fields is not inherently safer: every added dimension must distinguish independent events without splitting repeat deliveries.
5. Keep identity stable through representation changes
Resolve account, environment, and route mappings from authenticated, validated integration context. An arbitrary payload field should not choose its own trusted namespace. If the claimed account conflicts with the established connection, hold the event for investigation rather than choosing whichever value creates a convenient key.
Use constrained columns or an unambiguous structured representation. If storage requires a hash, derive it from canonical values and retain enough metadata to explain matches. Preserve identifier case unless the contract permits normalization. Do not default an unresolved environment to production or confuse an unknown mapping with a genuinely inapplicable dimension.
In this worksheet, key_format is representation metadata and does not participate in uniqueness. Changing its label must not create another accepted event. Changing serialization or hashing still requires compatibility: compare canonical identity columns, backfill the new representation before cutover, or use a reviewed dual-lookup period that finds existing records.
If a design instead includes a version inside the unique key, changing that version creates a new identity unless explicitly bridged. Require a compatibility lookup or reliable backfill that preserves matches for previously accepted events. Test an old receipt followed by its replay under the new version before release.
Apply uniqueness atomically. A separate lookup followed by insertion can let concurrent deliveries both proceed. Identity scope determines which records compete; the storage mechanism determines whether that rule survives concurrency.
6. Preserve required work when acceptance is interrupted
Suppose the hypothetical commerce pipeline must update an order and create an operations task. Accepting the event is not evidence that either action completed. Give each action a stable processing identity derived from the event identity and an action name, with completion tracked separately.
Where feasible, persist event acceptance and durable work intent in one atomic transaction. Otherwise, retain a recoverable event record containing the validated event data, or a durable reference, and enough information to determine its required actions. A reconciler must be able to recreate missing action records from that record even after later deliveries are classified as duplicates.
This closes a specific gap: the receiver accepts an event, stops before creating its actions, and subsequently suppresses its retry. A duplicate path should locate the existing event and permit reconciliation; it should not equate accepted with fully processed.
Test interruption immediately after acceptance and before action creation. The observable recovery outcome is that missing action records appear without accepting a second event. Also test interruption after an external action succeeds but before local completion is recorded. Reconciliation alone cannot guarantee exactly-once external effects; use destination-supported idempotency where available, or inspect uncertain outcomes before repeating an action.
Replacing an overlapping subscription should preserve receipt identity. Conversely, merging independent workflows must not erase either workflow’s outstanding obligations.
7. Repair an existing boundary without replaying everything
A wider key fixes future matching, not historical suppression. A wholesale replay may repeat actions that already succeeded. Use a controlled migration:
- Preserve evidence. Retain available deliveries, payload references, receipt decisions, and processing outcomes. Identify accounts and environments that shared the old namespace.
- Compare without execution. Calculate proposed identities in observation mode. Find old duplicate decisions that split into independent namespaces.
- Backfill reliable identities. Use established account and environment context. Mark records whose missing context prevents reliable reconstruction.
- Check actual outcomes. A suppressed delivery does not prove every downstream action is absent. Inspect processing records and relevant destination state.
- Bridge the cutover. Preserve matching for genuine old events. Do not reject every new scoped key merely because its unscoped identifier exists; that perpetuates the original collision.
- Recover reviewed work. Authorise replay or action recovery explicitly, recording why each item was released and which obligations remain.
Choose receipt retention separately. Scope determines which events match; retention determines how long that evidence remains available. Expired records cannot support a later duplicate match, regardless of how well the key was designed.
8. Verify outcomes before and after release
Attach identifier-scope evidence to the worksheet and list overlapping subscription pairs. Treat unresolved mappings as blockers, not blank values that silently compare equal. Run the main collision fixtures, independent-subscription fixtures, and type-local fixtures appropriate to the integration.
Add concurrent delivery, subscription replacement, old-format replay, and interrupted-processing tests. Require both sides of correctness: independent events are accepted, and repeat deliveries locate the existing event. Confirm that missing work can be recovered without inventing a new event identity.
Monitor accepted events, duplicate decisions, unresolved mappings, and unfinished actions by the dimensions chosen for the key. Keep representation version visible for migration diagnosis without assuming it belongs in identity. Investigate matches where incoming metadata conflicts with the stored event.
A lower duplicate count alone is inconclusive: it can indicate fewer false matches or more repeated acceptance. Start with one duplicate decision and explain why both receipts represent the same event under the documented contract. If the explanation relies only on identical strings while ignoring unresolved namespace boundaries, the design is not ready.
Put the next conversation to work.
Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.
Explore Communications