Drain an integration backlog without causing a second outage
Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.
The dependency is responding again, but thousands of queued tasks are waiting. Releasing them all can turn recovery into another incident: workers compete for connections, retries multiply, and live requests wait behind yesterday’s work.
To drain an integration backlog after an outage, first verify recovery with limited traffic. Then set a backlog budget below the dependency’s demonstrated capacity, reserve room for live work, and release queued tasks in stages. Increase the rate only while dependency health, live-request performance, and queue progress remain inside explicit limits.
The controlling number is not how many workers you can restart. It is how much additional work the entire request path can absorb. The filled fictional runbook below shows how to turn that distinction into a release plan.
1. Establish what recovery actually means
A successful health check proves less than a successful business request. Before draining, verify the path queued tasks will actually use: authentication, the relevant operation, connection handling, and any downstream processing that determines completion.
Keep these checks narrow and authorised. Prefer a non-mutating check where it represents the affected path; if it does not, use a small, explicitly authorised set of already-eligible tasks. Do not create broad production activity merely to prove that production is available.
Record a recovery baseline before adding meaningful backlog traffic:
- Dependency: response latency, failures, timeouts, rate-limit responses, and any visible saturation signal.
- Local infrastructure: worker utilisation, connection-pool occupancy, internal queue delay, and resources shared with live requests.
- Live work: arrival rate, completion rate, end-to-end latency, and errors.
- Backlog: eligible count, oldest eligible task age, attempt rate, and confirmed completion rate.
Use a sustained observation window, not one successful request. A service can answer probes while its processing queue continues to grow. If accepted requests are asynchronous, distinguish acceptance from completion and watch both.
This procedure starts after work has been judged eligible to run. Uncertain previous writes, stale-task decisions, and campaign reconciliation need separate resolution; a faster drain does not resolve them.
2. Calculate a budget in dependency attempts, not worker count
Choose a unit that reflects the bottleneck. If the dependency is constrained by requests per second, count every dependency attempt, including retries. If payload size or operation cost varies substantially, use separate budgets for expensive operations or a conservative cost-weighted budget.
A useful starting rule is:
Backlog attempt budget = max(0,
conservative dependency capacity
- live-work reservation
- safety headroom)
Actual release rate = the lowest applicable budget
across the dependency and shared local resourcesA configured quota is a ceiling, not evidence that the recovering service can currently sustain it. Base the initial estimate on recent comparable healthy operation, recovery observations, and any verified provider guidance. Where that evidence is weak, begin with a small probe stage rather than treating the estimate as established capacity.
Reserve for plausible live demand, not just the latest quiet minute. Keep separate headroom for variability and imperfect estimates. These reservations reduce the advertised drain speed, but make the plan less dependent on live traffic staying unusually low.
Apply the budget globally across workers that share the constrained resource. A per-worker limit becomes a much larger aggregate limit when additional workers start. Also bound concurrency: an attempt-rate limiter alone does not prevent slow requests from accumulating in flight.
3. Filled fictional runbook: capacity, ownership, and release gates
Fictional worked example: All quantities, timings, thresholds, and outcomes in this runbook are hypothetical operating choices, not provider limits, benchmarks, or customer results. The example assumes one comparable dependency request per task and no ordering constraint between the selected tasks.
Recovery record and initial estimate
- Incident: a CRM synchronisation dependency has recovered after an interruption. There are 72,000 eligible tasks waiting; the oldest is 80 minutes old.
- Authority: the incident lead authorises each increase. One drain operator changes the release setting; a second operator watches live traffic and dependency health.
- Observed live demand: 45 dependency attempts per second during the recovery baseline.
- Conservative total-capacity estimate: 120 attempts per second, informed by comparable healthy operation and checked cautiously during recovery. This remains provisional.
- Live-work reservation: 60 attempts per second, leaving room above observed demand.
- Safety headroom: 20 attempts per second, not allocated to routine draining.
- Maximum backlog budget: 120 − 60 − 20 = 40 attempts per second, including backlog retries.
- Concurrency control: start with at most 10 backlog attempts in flight; never exceed 40 during this run. Live traffic uses protected execution capacity.
- Recovery baseline: observe the relevant dependency path and live traffic for 10 minutes before starting the staged drain.
Release schedule
- Stage A: 5 backlog attempts per second for at least 5 minutes. Keep backlog concurrency at 10. Confirm that eligible tasks complete, not merely leave the dispatch queue.
- Stage B: 10 attempts per second for at least 5 minutes. Retain the same concurrency cap unless measurement shows it is unnecessarily restricting healthy completions.
- Stage C: 20 attempts per second for at least 5 minutes. Raise backlog concurrency to at most 20 only if needed and approved.
- Stage D: 30 attempts per second for at least 5 minutes. Allow at most 30 backlog attempts in flight. Check live-demand forecasts before the next increase.
- Stage E: up to 40 attempts per second. Allow at most 40 in flight and review every 5 minutes until the backlog clears or conditions change.
Each stage is a ceiling, not a target that must be reached. Do not add workers to compensate for a concurrency cap that is binding because requests have slowed.
Conditions required to advance
- Live-request p95 end-to-end latency stays at or below 800 milliseconds.
- Dependency p95 response latency stays at or below 400 milliseconds.
- Dependency timeouts and server errors together stay below 1% of attempts.
- No rate-limit responses occur during the stage.
- Live arrivals remain within the 60-attempt-per-second reservation.
- Confirmed backlog completions increase without a growing local or downstream processing queue.
- Shared connection-pool occupancy stays below 70%.
The operator records actual attempts, completions, retries, queue age, and each health measure before requesting the next stage. Missing or unreliable measurements mean hold, not advance.
4. Protect live work with enforcement, not intention
A spreadsheet reservation does nothing if backlog workers occupy every connection. Enforce the separation at the shared bottlenecks: scheduling, worker slots, connection pools, or whatever resources the architecture actually shares.
In the fictional runbook, unused live capacity is not automatically loaned to backlog work. That sacrifices some speed but keeps the first recovery plan easy to reason about. Borrowing can be considered later only if it can be withdrawn promptly without leaving a large amount of backlog work in flight.
Recalculate the budget when live demand changes. If hypothetical live demand rises to 75 attempts per second, the backlog ceiling becomes 120 − 75 − 20 = 25 attempts per second. A drain running at 40 must step down; the earlier approval does not override current conditions.
Check fairness within the backlog too. One tenant, workflow, or costly task type should not consume the entire budget. Use bounded shares where needed, and preserve any required per-record ordering. Oldest-first scheduling is useful only where tasks are independent or the ordering rules explicitly permit it.
For outbound communications, task eligibility also depends on current permission and channel requirements. Keep that eligibility decision separate from release pacing; a slot becoming available is not authorisation to send.
5. Treat retries as load and rate limits as feedback
A common drain failure is limiting original dispatches while allowing retries to bypass the limit. The reported drain rate looks controlled, but the dependency sees more attempts than the plan permits.
Put initial attempts and retries through the same aggregate budget. Use bounded retries with backoff and jitter, and keep future retries visible in the workload estimate. When failures rise, reduce offered load rather than spawning replacement attempts faster.
An HTTP 429 response indicates rate limiting; it may include a Retry-After value. Providers apply limits differently, so check whether the affected limit covers an account, operation, resource, or another scope. Honour applicable retry instructions and avoid continuing the same workload through a different worker.
Do not keep workers occupied sleeping through long retry delays if the queue design can safely schedule a later attempt instead. Conversely, do not return delayed work to an immediately runnable queue and create a tight retry loop. The goal is less pressure on the dependency, not merely a different place to hold it.
6. Define pause and rollback before releasing the queue
The runbook needs actions attached to signals. “Watch closely” leaves the hardest decision until operators are under pressure.
Hypothetical pause and rollback rules for the worked example:
- Hold the current stage: any required metric is unavailable, or there is not yet a full healthy 5-minute observation window. Do not increase the rate.
- Return to the last healthy stage: dependency p95 latency exceeds 400 milliseconds or connection-pool occupancy exceeds 70% for two consecutive 1-minute windows, while live work remains healthy.
- Pause new backlog attempts immediately: live p95 latency exceeds 800 milliseconds for two consecutive 1-minute windows; dependency timeouts and server errors reach 1% for two such windows; any rate-limit response appears; or a dependency failure is confirmed.
- Pause for loss of control: the global limiter is not enforcing its budget, retries bypass it, or downstream pending work grows while apparent dispatch throughput stays high.
- Resume only with approval: after the applicable retry delay has elapsed, the cause has been checked, and a fresh 10-minute healthy baseline is available. Restart at Stage A rather than the interrupted rate.
Here, rollback means reducing release pressure. Stop leasing new backlog work and prevent automatic catch-up bursts. Let already-started requests finish where safe; do not blindly cancel operations that may already have taken effect. Any resulting uncertain outcomes leave this drain procedure for separate verification.
If live traffic alone exceeds available capacity, pausing the backlog is necessary but insufficient. Return to incident handling for the live path rather than repeatedly restarting the drain.
7. Measure useful progress and finish deliberately
A falling dispatch-queue count can hide work moving into another queue. Track confirmed completion rate alongside pending asynchronous work, retry volume, and the age of the oldest eligible task. Investigate a stationary oldest task even when the total count falls: it may be blocked, repeatedly failing, or starved by scheduling.
For the fictional schedule, assuming immediate completions and no retries, the first four 5-minute stages complete 19,500 tasks. The remaining 52,500 tasks take about 22 minutes at 40 per second. That gives an optimistic total of roughly 42 minutes from the start of Stage A, excluding the recovery baseline.
This is a planning estimate, not a deadline. Retries consume slots without necessarily completing additional tasks; slower responses can make the concurrency cap bind; live demand can reduce the backlog budget. Refresh the estimate from observed net completions and remaining eligible work rather than pushing the rate to defend the original finish time.
Close the drain only when the eligible backlog and associated in-flight work are accounted for, deferred retries have an owner, and live traffic remains healthy after temporary recovery settings are removed. Record the highest healthy stage, any pause, and the constraint that actually limited progress.
For the next recovery, prepare one filled release record before enabling workers: capacity evidence, live reservation, initial rate, concurrency cap, stage duration, pause thresholds, and the person authorised to advance. The first measurable success is a complete observation window in which backlog completions rise without degrading live work—not the fastest possible fall in queue count.
Put the next conversation to work.
Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.
Explore Communications