Back to all articlesIntegration reliability

Control intake when an integration cannot keep up

Alactic Editorial10 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

When new work keeps entering an integration faster than it completes, a queue becomes a growing business commitment. Requests may already be accepted, but their deadlines, storage needs, and eventual recovery effort continue to accumulate.

The practical response is to control admission before those commitments become unmanageable. Slow intake when callers can wait or retry safely. Defer work when its business value survives a delay. Reject requests visibly when you cannot responsibly accept them. Pause a source when it cannot respect those controls or when accepting more work would deepen an incident.

A useful integration queue backpressure strategy defines those choices in advance, gives every accepted task a visible state, and specifies who can restore normal intake. The worked decision table below shows how to make that policy operational without assuming any particular product controls.

1. Measure the waiting commitment, not just queue size

Queue depth tells you how many tasks are waiting, but not whether the business is already in trouble. A small queue of expensive tasks can be worse than a large queue of quick ones. A stable count can also hide urgent tasks stuck behind less important work.

Use a compact set of signals for each meaningful work class:

  • Oldest eligible task age: how long the oldest task that should be runnable has waited. Separate intentionally deferred tasks from tasks waiting for execution.
  • Deadline risk: whether accepted work is approaching its useful completion time.
  • Arrival and completion rates: whether work is entering faster than it leaves, measured over the same interval and in comparable work units.
  • Deferred-work age: how long postponed work has been waiting for release or review.
  • Storage headroom: whether the remaining space can absorb expected intake and operational records during recovery.

Inspect these signals by source and priority as well as in aggregate. Otherwise, a healthy bulk-import queue can conceal a stalled urgent queue. Keep failed attempts distinct from completed tasks: a busy worker repeatedly failing the same operation is not draining the backlog.

Queue age should trigger investigation and admission decisions, not act as the only safety check. A sudden storage risk or repeated downstream failure may justify a pause before the age threshold is reached.

2. Choose the control that matches the business consequence

These controls are not interchangeable. Each transfers a different cost to the caller, the business team, or the recovery process.

Slow intake when demand can adjust

Reduce the rate at which new work is admitted while preserving room to finish accepted tasks. This suits callers that can pace submissions and retain their own pending work. It fails when callers interpret a delay as a lost request and send duplicates, or when every source independently retries at once.

Defer when later completion still has value

Accept nonurgent work into a separately visible deferred state with a release condition and an expiry or review rule. Deferral is a commitment to manage work, not permission to hide it. If deferred items share the same constrained storage, they still consume headroom and need an admission ceiling.

Reject when acceptance would mislead

Reject before taking responsibility for execution when you cannot provide a reasonable processing path. Tell the caller that the request was not accepted, whether a retry is appropriate, and how to seek help. Rejection can interrupt a business process, but a clear refusal can be safer than an apparent success followed by an indefinite wait.

Pause a source when finer controls are insufficient

Pause a bulk producer, scheduled import, or other controllable source when it is overwhelming the integration or ignoring feedback. Identify where unsent work remains and how it will be reconciled later. A pause without that record can create an invisible gap that is harder to recover than the original backlog.

3. Use a filled backpressure decision table

Fictional worked example: A business integrates customer updates and scheduled data enrichment. Customer updates have higher urgency; enrichment can wait. All values below are hypothetical operating choices, not benchmarks. The team checks queue signals every five minutes and assigns an integration on-call owner, a business operations owner, and a source owner.

The following decision table is presented as row-by-row records so it can be copied into a runbook. Stronger controls remain available when storage, failures, or deadline risk require them, even if the age signal is lower.

  • Normal admission.
    Queue-age signal: Oldest eligible customer update is under two minutes, with no accepted deadline at risk.
    Intake control: Admit within the agreed source budgets; keep enrichment in its own work class.
    Caller feedback: Return an acceptance reference and a pending state, not a completion claim.
    Responsible owner: Integration on-call monitors; business operations owns priority definitions.
    Restoration condition: This is the normal state. Move to a stronger control if age grows or another safety signal trips.
  • Slow admission.
    Queue-age signal: Oldest eligible update is two to five minutes and age is rising across two checks.
    Intake control: Cap new customer-update admission at a hypothetical 60 tasks per minute against a recently observed healthy completion rate of 90 comparable tasks per minute; stop increasing any source budget.
    Caller feedback: Explain temporary rate restriction and the retry procedure; unaccepted requests remain the caller's responsibility.
    Responsible owner: Integration on-call applies the cap; source owners confirm pacing.
    Restoration condition: Age stays below two minutes across three checks, completions exceed admissions, and storage headroom is stable.
  • Defer nonurgent work.
    Queue-age signal: Oldest eligible update exceeds five minutes, or continuing enrichment would threaten update deadlines.
    Intake control: Stop releasing enrichment for execution. Accept it as deferred only within a hypothetical 10,000-item holding ceiling, with review after four hours.
    Caller feedback: Show deferred status, reference, review time, and instructions not to resubmit.
    Responsible owner: Business operations approves deferral; integration on-call controls release.
    Restoration condition: Urgent age remains below two minutes across three checks; release a limited enrichment batch without reversing that trend.
  • Reject new requests visibly.
    Queue-age signal: Oldest eligible update exceeds ten minutes, or the deferred ceiling is reached. Unsafe storage headroom can trigger this earlier.
    Intake control: Refuse new lower-priority requests before acceptance; protect the processing path for already accepted work.
    Caller feedback: State not accepted, give a retry instruction when appropriate, and provide an escalation route for genuinely urgent cases.
    Responsible owner: Incident lead authorises the restriction; integration on-call implements it; business operations handles exceptions.
    Restoration condition: Age falls below five minutes across three checks, storage is stable, and the team can admit a limited batch without displacing accepted work.
  • Pause the bulk source.
    Queue-age signal: Age keeps rising despite restrictions, or the source continues submitting beyond its allowance. A critical storage alert can trigger an immediate pause.
    Intake control: Stop the scheduled enrichment source at its producer; preserve a checkpoint of the last confirmed accepted item.
    Caller feedback: Mark the source paused, identify the accountable owner, and distinguish unsent items from accepted backlog.
    Responsible owner: Source owner executes the authorised pause; incident lead records the decision.
    Restoration condition: The failure is contained, the checkpoint is reconciled, and a small restart batch completes without renewed age growth.

These rows are not a mandatory staircase. A producer flooding an unavailable dependency may need an immediate pause. Conversely, slow intake and deferral may operate together while urgent work continues.

4. Make acceptance, deferral, and rejection unmistakable

The most costly ambiguity is telling a caller that a request succeeded when only receipt succeeded. Define the acceptance boundary: the point at which the integration has a durable record and takes responsibility for tracking the task to an explicit outcome.

For accepted work, retain a reference, source, acceptance time, priority, current state, and any review or expiry condition. Offer a status-check path appropriate to the integration. If acceptance cannot be confirmed after a connection failure, callers need a way to reconcile that uncertainty before resubmitting.

For rate-limited HTTP callers, HTTP 429 indicates rate limiting, and a Retry-After header may communicate when to retry. Providers apply limits differently, so test the caller's actual response rather than assuming it will wait. Do not use a rate-limit response to disguise a request that has already been accepted.

Hypothetical feedback examples:

  • Deferred: “Accepted as E-184. Enrichment is deferred during recovery. Next review is at 14:00. Do not resubmit this item.”
  • Rejected: “Not accepted because intake is temporarily restricted. Retain this request and retry after the indicated interval. Contact operations if its business deadline cannot wait.”
  • Paused source: “Scheduled intake is paused. Accepted items remain tracked; later source items have not yet been submitted.”

Provide a human escalation route, but do not let an escalation bypass capacity controls automatically. Someone must assess which existing work an exception would delay.

5. Prevent retries and deferrals from becoming a second overload

Backpressure fails if it merely moves the same load into another queue with less visibility. Include retry traffic in the intake budget. Ask callers to use bounded retries with increasing delays and variation between attempts, rather than immediate synchronized resubmission.

Use a consistent request identifier and a documented duplicate-handling rule where the integration supports them. A timeout after acceptance must not automatically become a second business action. If duplicate handling is unavailable, make reconciliation an explicit recovery step rather than claiming retries are safe.

Deferred work needs its own inventory. Record how much is waiting, its oldest age, its business owner, and what will happen when it becomes stale. At review, the owner may authorise execution, revise the task, or explicitly cancel it with a recorded reason. Never silently discard an accepted task.

Priority also needs limits. If every source can label its own work urgent, the urgent lane becomes the ordinary queue. Restrict priority changes to authorised owners and review exceptions during the incident. Leave a bounded share of processing for older lower-priority work when capacity permits, so deferral does not become permanent starvation.

6. Restore intake in stages, not with one switch

A falling queue is evidence of progress, not proof that the system can absorb a full restart. Recovery must account for deferred work, paused-source catch-up, and clients waiting to retry.

  1. Reconcile commitments. Count accepted pending tasks, deferred tasks, uncertain submissions, and unsent source items separately.
  2. Confirm useful progress. Check completed business operations, not merely successful worker starts or reduced error messages.
  3. Reopen a limited slice. Release one source or work class within a defined admission budget. Keep enough processing headroom to continue draining accepted work.
  4. Observe through a full check period. Watch age, completion rate, storage, and downstream failures before raising the budget.
  5. Roll back if the signal reverses. Reapply the previous restriction when age rises or failures recur; record what changed.

For the hypothetical example, normal intake requires three consecutive healthy checks, but each source still returns gradually. Restarting enrichment, all paused imports, and all waiting retries together would defeat the purpose of the recovery gate.

Before executing stale work, check whether it still represents the intended business action. A newer customer update may supersede an older one, but supersession must follow an agreed rule and leave an explicit outcome for the original task. It is not a justification for silently dropping backlog.

7. Turn the policy into a measurable operating check

Start with one integration and one source that can be controlled. Assign an owner to every row of the decision table, confirm who may change admission settings, and document what callers will actually see. These are recommended operating practices; verify which controls and status information your current workspace and plan support before relying on them.

Run an authorised, bounded rehearsal using a fictional workload. Include a caller that retries after an ambiguous timeout, a deferred item that reaches review time, and a paused source that needs checkpoint reconciliation. Confirm that none is mistaken for completed work or lost between owners.

The next action is concrete: capture a baseline of eligible queue age, arrivals, completions, deferred age, and storage headroom, then fill your own thresholds and restoration gates. Judge the policy by whether age stabilises after restriction, every accepted item remains accountable, and reopening intake avoids another rise in waiting time. If one of those outcomes cannot be observed, improve the state records or caller feedback before relying on the policy during a larger overload.

Integration reliabilityBackpressureQueue managementOperations

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications