Choose which integration failures deserve a retry
Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.
A rejected credential should not keep returning to the same queue as a dropped connection. Yet a rule such as “retry whenever the request fails” treats both as problems another attempt might solve. The result is avoidable traffic, repeated alerts, and a backlog that hides the failures someone actually needs to fix.
Retry an API error only when the failure is plausibly temporary, repeating the operation is safe, and the work is still valid. Stop unchanged attempts when credentials, permissions, or input must be corrected. When the request may already have completed, establish its outcome or establish that repetition cannot create an additional effect before retrying.
The useful output is not a longer list of error codes. It is a decision record: what failed, what evidence supports the classification, what action is allowed, and what ends that action. The fictional decision table below provides a starting point. Retry timing, credential rotation procedures, payload mapping, and individual providers’ error meanings are separate decisions.
1. Classify the failure and the completion state separately
Use three operational classes. They describe what should happen next, rather than claiming to identify every underlying cause.
- Transient: the condition could clear without changing the intended request. Repetition may be eligible, subject to completion safety and a finite retry allowance.
- Persistent: the available evidence indicates that repeating the unchanged request will not help. A credential, permission, input, configuration, or business decision must change first.
- Uncertain: the evidence is insufficient to choose safely. This includes cases where the integration does not know whether an operation took effect.
Persistent does not mean permanent. A rejected request can become eligible after a verified correction. Transient does not mean automatically retryable. A temporary connection problem can still leave a completed write behind.
Record completion separately as not applied, applied, or unknown, based on the evidence available. This prevents a common mistake: interpreting “the client did not receive success” as “the remote system did nothing.” A failure response, an absent response, and a confirmed rejection are not interchangeable.
2. Use a filled retry decision table
Fictional worked example: the following table describes a hypothetical integration that synchronises business records and submits authorised outbound actions. Every classification, evidence requirement, and stopping rule is a proposed design—not a claim about a particular product or provider.
The compact table is followed by the evidence needed to use each row. “Eligible” means permitted to enter a bounded retry process, not instructed to repeat indefinitely.
Failure Class Action Stop condition Connection interruption Transient* Retry only if safe Success, cap, or ambiguity Dependency overload Transient* Gate affected requests Success, cap, or stale work Rejected credentials Persistent Block unchanged retries Verified correction required Invalid input Persistent Return for correction Corrected, revalidated work Unknown completion Uncertain Reconcile before replay Outcome found or review needed *Provisional: completion safety must also be established.
Connection interruption
Evidence needed: transport diagnostics showing where the interruption occurred; whether the request could have reached the dependency; the operation’s documented repetition behaviour; and any operation reference or status evidence. If the client can reliably establish that nothing was sent, that supports retry eligibility. If transmission began and completion cannot be established, use the unknown-completion row.
Action: permit a bounded retry when the operation was not applied or repetition is demonstrably safe. For a state-changing operation, require a verified deduplication mechanism or another documented safety property before replaying an ambiguous request.
Stop: stop on confirmed success, exhausted retry allowance, expired business relevance, or evidence that the outcome is uncertain. Do not let a transport label override those conditions.
Dependency overload
Evidence needed: a documented response indicating temporary capacity pressure or rate limiting, plus enough context to identify the affected dependency, account, or operation. A status code alone should not establish whether a write occurred.
Action: gate affected work and allow only requests that remain valid and safe to repeat into the bounded retry process. Keep unrelated work moving where the evidence supports a narrower failure boundary.
Stop: stop on success, exhausted allowance, work expiry, or evidence that the condition actually requires a configuration or entitlement change. Reclassify rather than treating every continued rejection as temporary overload.
Rejected credentials
Evidence needed: a documented authentication rejection, the identity of the affected connection, and confirmation that the result is not merely an unmapped response. Record a safe credential reference or version identifier, not a secret.
Action: block unchanged attempts using the affected credentials and route the issue to the connection owner. Release work only after an authorised correction has been verified for the intended operation.
Stop: the automatic retry path stops at classification. A verified correction can create a new eligibility decision; it does not justify endlessly replaying the original rejection.
Invalid input
Evidence needed: an understandable validation response tied to the submitted request version, with the rejected field or rule identified where available. Distinguish a data rejection from an unreadable or unexpected error response.
Action: return the item for correction. Keep it out of unchanged retries. After correction, validate the new version and confirm that the underlying business action is still wanted.
Stop: stop automatic attempts until corrected input passes the applicable checks. If correction is unavailable or the work is no longer needed, close it with an explicit disposition rather than leaving it pending.
Unknown completion
Evidence needed: a stable operation reference, a documented status lookup or reconciliation method where available, and evidence of any deduplication guarantee. Establish its scope and conditions rather than assuming a repeated identifier makes replay safe.
Action: hold the item for reconciliation. Confirmed completion closes it without replay. Confirmed non-completion permits a fresh eligibility check. A verified safe-repetition mechanism may permit replay, but only within its documented conditions.
Stop: if the outcome remains unresolved and safe repetition cannot be established, stop automatic execution and assign human review. Do not convert an unresolved outcome into permission to retry.
3. Treat status codes as evidence, not the whole policy
Status codes are useful inputs to classification, but they do not answer every operational question. The same broad response category can appear across operations with very different consequences if repeated.
For example, HTTP 429 indicates rate limiting, and a response may include a Retry-After header. Providers differ in how they apply limits. This supports investigating a temporary limit; it does not, by itself, establish that the underlying action was not applied or that a particular replay is safe. Timing instructions belong in the scheduling policy after eligibility is established.
Likewise, do not create universal rules such as “retry every server error” or “never retry any client error.” Use the provider’s current documentation to interpret the response, then combine it with operation type and completion evidence.
A useful rule record includes the error signature, affected operation, evidence required, classification, permitted action, and owner. If a response is missing, contradictory, or undocumented, classify it as uncertain instead of inventing a meaning.
4. Make unknown completion a separate queue
A failed attempt and a failed business action are different observations. Keeping uncertain outcomes in the ordinary retry queue encourages the system to resolve uncertainty by trying again—the very action that may create a duplicate.
Fictional worked example: an integration submits a request to create a business record. The connection closes before the client receives a response. The remote system might have created the record before the interruption.
If the integration repeats the creation solely because it saw a connection error, it may create another record. Instead, the item enters an uncertain-outcome state with its original operation reference and request version intact. A documented lookup confirms whether the record exists. If it exists and matches the intended action, the integration records completion. If non-completion is established, it evaluates retry eligibility again.
If no reliable lookup exists, a supported deduplication mechanism may provide a safe route—but only if its documented coverage applies to this request. Generating a fresh reference for the replay can defeat that protection. If neither reconciliation nor safe repetition is available, human review is the stopping condition, not a design failure to conceal.
5. Define what must change before blocked work returns
“Needs attention” is not an actionable destination. Every persistent or unresolved item needs an owner, a reason it is blocked, and a release condition.
- Credential rejection: the connection owner verifies an authorised correction and appropriate access for the intended operation.
- Invalid input: the data owner supplies a corrected request version that passes validation.
- Uncertain outcome: the operations owner provides completion evidence or approves a documented resolution through the organisation’s normal authorisation process.
Do not repair authentication failures by weakening permission checks or broadening access simply to make a retry succeed. OWASP’s authorization guidance recommends least privilege, deny-by-default behaviour, and checking permissions on every request. A retry should remain subject to those checks.
Also recheck whether the action is still valid. A technically corrected request can be obsolete because a user cancelled the task or the business state changed. In communications workflows, current recipient permission and opt-outs remain gates; a past failed attempt is not continuing permission to send.
6. Put a finite exit on every eligible retry
A transient classification is a working hypothesis. If the condition does not clear, the system needs an exit that preserves both capacity and accountability.
Define the stopping rules before releasing eligible work:
- Completion: record success and remove the item from retry eligibility.
- Finite allowance: stop automatic attempts when the assigned allowance is exhausted and route the item to an owner.
- Business expiry: stop when the action is no longer useful or authorised.
- New evidence: reclassify when a response reveals invalid input, rejected access, or uncertain completion.
The allowance’s size and scheduling belong to separate capacity and timing decisions. The classification requirement is simpler: every retryable item must have a finite exit, and an operator must be able to explain why it remains eligible.
There is a trade-off. Conservative classification may delay legitimate work and increase review effort. Aggressive classification can consume capacity and repeat side effects. Use narrower evidence-based rules to reduce both costs rather than forcing all failures into one default.
7. Preserve enough evidence to recover
Store a compact decision record alongside the failed item. Include the operation reference, request version, affected dependency, safe error summary, completion state, classification rule, attempts consumed, owner, and next permitted action. Avoid retaining credentials or unnecessary sensitive payload content.
Keep request versions distinct. If someone corrects an invalid field, the corrected request should be recognisable as changed work linked to the original failure. Otherwise, reviewers cannot tell whether the system retried unchanged input or submitted an authorised correction.
Recovery also needs a release check. Do not return an entire blocked backlog merely because one test succeeds. Confirm that the correction applies to the affected group, then evaluate the remaining items for current relevance and completion safety. A broad release is convenient, but it can replay obsolete work or mix unrelated causes.
8. Test the classification against a real failure sample
Start with a bounded review rather than rewriting every integration rule at once. Hypothetical review exercise: select 30 recent failed operations from one integration. For each, fill in the decision table’s class, completion state, evidence, next action, owner, and stopping condition. The sample size is illustrative, not a benchmark.
Measure three observable outcomes:
- How many unchanged attempts would have been blocked because correction was required?
- How many ambiguous operations would have entered reconciliation instead of replay?
- How many items still lack enough evidence to make an eligibility decision?
Then implement one narrowly defined rule and review its decisions before enabling automatic retries under it. Track classification changes, duplicate effects where observable, exhausted allowances, and unowned blocked items. A falling retry count alone is not proof of improvement: useful work may simply be stranded.
The immediate next action is to take one recurring failure signature and document its evidence requirement, permitted action, and exit. If those cannot be written clearly, keep it out of automatic retry until the missing evidence or ownership is resolved.
Put the next conversation to work.
Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.
Explore Communications