Back to all articlesIntegration reliability

Diagnose queue growth before adding more workers

Alactic Editorial10 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

A queue is growing, so someone proposes adding workers. Before approving that change, establish whether more work is arriving, successful work is leaving more slowly, consumers have stopped, or failed attempts are cycling back into the queue. Similar queue charts can require very different responses.

Start with aligned observations of arrival rate, successful completion rate, oldest unresolved-task age, attempt duration and dependency health. Keep queue entries separate from unique business tasks: a repeatedly retried task can generate substantial activity without representing new demand.

The goal is not to rule out capacity changes. It is to identify the constraint first. The filled worksheet below demonstrates four fictional patterns and gives a next observation for each.

1. Define what the queue counter counts

Before interpreting a rising line, write down its boundary. Does the counter include only entries ready to be claimed, or also work already being processed? Does it include delayed retries? Are failed entries removed, moved elsewhere or left pending? A dashboard label such as “pending” does not answer these questions.

Maintain two accounting views:

  • Queue-entry view: entries added, claimed, removed, waiting, in flight and awaiting retry. This explains transport activity and storage pressure.
  • Business-task view: unique tasks accepted, successfully completed, terminally failed and still unresolved. This explains whether useful work is progressing.

Within a defined boundary, the change in entry count equals entries added minus entries removed. Removal is not necessarily success: a worker might release an entry after scheduling a retry or recording a permanent failure.

Unresolved business tasks increase with newly accepted tasks and decrease with successful completions or explicit terminal dispositions. Retries do not create new business demand, even when they create new entries.

Use a stable task identifier across retries and record attempt identifiers separately. If that link is missing, label the diagnosis provisional rather than interpreting attempt counts as unique demand.

2. Capture one aligned diagnostic snapshot

Collect these observations for the same queue, route and interval. Compare them with a healthy period handling reasonably similar work. Whole-system averages can hide one blocked destination behind several healthy ones.

  • New-task arrivals: unique business tasks accepted per minute. Record total queue-entry arrivals, including retries, separately.
  • Successful completions: unique tasks reaching the intended outcome per minute, not worker acknowledgements or request starts.
  • Oldest unresolved-task age: elapsed time since original acceptance, including in-flight time and scheduled retry delays. Also inspect the oldest ready-entry age.
  • Attempt duration: elapsed time from claim to attempt outcome. Separate local execution from dependency waiting within that duration.
  • Scheduled retry delay: waiting between attempts. Measure this separately; it contributes to end-to-end task age, not the completed attempt’s claim-to-outcome duration.
  • Dependency health: response latency, timeouts, error categories and rate-limit responses.
  • Consumer activity: recent claims, in-flight work, active consumers and time since the last successful completion.

Use both a central measure and a tail measure for attempt duration, such as median and p95. A stable median can conceal a minority of blocked attempts. Completed-attempt timing also excludes attempts that have not finished, so inspect current in-flight ages directly.

Record the start and end of the observation window. With a consistent measured population and original-acceptance timestamps, an existing task ages only by the elapsed interval. An unexpectedly larger jump can indicate that older work entered the scope or that the measurement changed.

3. Use the filled queue-diagnosis worksheet

Fictional worked example: an integration accepts business tasks, queues attempts and calls a downstream service. Each scenario below uses a hypothetical ten-minute window. Queue depth includes ready, in-flight and delayed-retry entries. All numbers are illustrative, not benchmarks or customer results.

Reuse the field structure for an incident: queue movement; arrivals; successful completions; age and duration; dependency and consumers; working diagnosis; next observation.

Pattern A: traffic burst with healthy processing

  • Queue movement: entries rise from 300 to 900.
  • Arrivals: 180 new tasks per minute versus a healthy baseline of 100. Total entry arrivals are also 180 per minute; no retries are observed.
  • Successful completions: 120 tasks per minute, unchanged from the healthy processing rate.
  • Age and duration: oldest unresolved age rises from 3 to 6 minutes; attempt p95 remains 2 seconds.
  • Dependency and consumers: dependency timing and errors remain near baseline; consumers continue claiming and completing work.
  • Working diagnosis: new demand exceeds current throughput.
  • Next observation: check whether arrivals fall below completions in the next aligned window and depth begins shrinking.

Worked calculation: 180 arrivals minus 120 successful completions leaves 60 additional tasks per minute. Over ten minutes, that is 600 additional tasks, matching the increase from 300 to 900 entries. This reconciliation assumes no retries, terminal failures or other entry removals in this hypothetical window.

Pattern B: dependency slowdown

  • Queue movement: ready and in-flight entries increase; delayed retries begin accumulating.
  • Arrivals: new demand remains at 100 tasks per minute. Total entry arrivals are higher because some attempts are retried.
  • Successful completions: fall from a healthy 110 to 35 tasks per minute.
  • Age and duration: oldest unresolved age rises from 2 to 12 minutes; attempt p95 rises from 2 to 18 seconds.
  • Dependency and consumers: downstream latency and timeout counts increase. Consumers remain active, but many attempts wait on the same service.
  • Working diagnosis: dependency waiting restricts useful throughput. More consumers could add pressure without increasing successful work.
  • Next observation: separate local execution from dependency waiting and compare affected and unaffected routes. Check whether dependency waiting explains the longer attempts.

Pattern C: worker failure or stalled consumption

  • Queue movement: ready entries climb while new claims nearly stop; several old in-flight entries remain.
  • Arrivals: 100 new tasks per minute, with little retry traffic.
  • Successful completions: drop from a healthy 105 per minute to almost none.
  • Age and duration: oldest unresolved age reaches 20 minutes. Completed-attempt p95 still displays 2 seconds, mostly from samples preceding the failure.
  • Dependency and consumers: unrelated dependency traffic appears healthy. Consumer heartbeats are missing, or consumers report running without making claims.
  • Working diagnosis: work is not reaching execution, or existing execution has stalled. Historical duration samples do not establish current health.
  • Next observation: inspect each consumer’s last claim, last completion and current in-flight age. Locate the break before or after claiming, then inspect the relevant error or wait state.

Pattern D: retry loop masquerading as demand

  • Queue movement: entry activity and delayed retries rise sharply; the same task identifiers recur.
  • Arrivals: 40 new tasks per minute but 240 total entry arrivals, including 200 retry entries.
  • Successful completions: only 5 unique tasks per minute despite high attempt throughput.
  • Age and duration: oldest unresolved age reaches 45 minutes; most attempts fail in under a second.
  • Dependency and consumers: consumers remain active. Repeated attempts show the same rejection category, such as invalid input or a permission failure.
  • Working diagnosis: attempt volume is being mistaken for demand. Short durations reflect fast failures rather than efficient completion.
  • Next observation: group attempts by task identifier and error category. Check whether anything changes between attempts that could permit success, and whether a stopping condition exists.

4. Use age to challenge healthy-looking throughput

A completion chart can conceal stranded work. Consumers may keep finishing newer, easier tasks while older tasks remain unresolved. Stable aggregate depth does not establish that every task is progressing.

Sample the oldest unresolved tasks. For each, record original acceptance, current state, last attempt, next eligible retry time and last failure. Look for a shared destination, task type or dependency rather than assuming the whole queue has one cause.

Compare ready-entry age with original task age. A retry may create a fresh entry timestamp while the business task has already waited much longer. Reporting only entry age can make old work appear new.

Separate intentionally scheduled work from work already eligible for processing. A future execution time is not itself evidence of a stall, but original acceptance time still matters when assessing how long the business task has remained unresolved.

5. Read dependency errors before increasing concurrency

Dependency problems are not interchangeable. Distinguish the signals before choosing an intervention:

  • Slow responses: determine whether dependency waiting accounts for most of the increased attempt duration.
  • Timeouts: investigate whether the remote operation could have completed despite the missing local confirmation.
  • Repeated input or permission failures: inspect the task and configuration. Repeating an unchanged rejection does not demonstrate a capacity shortage.
  • Rate-limit responses: inspect the limit signal and retry behaviour before adding callers.

HTTP 429 indicates rate limiting. A response may include a Retry-After header indicating when to retry. Providers apply limits differently, so the status alone does not establish a universally safe concurrency level.

Break observations down by destination or operation where possible. One troubled route can dominate growth while others remain healthy. A system-wide worker increase may obscure that distinction and increase pressure on the affected dependency.

6. Choose an observation that could disprove the diagnosis

The worksheet’s next observations are tests, not confirmations to collect selectively. Write down what would weaken your preferred explanation. For example, if dependency waiting remains stable while local execution slows, investigate local processing rather than retaining the dependency diagnosis.

Missing telemetry is uncertainty, not evidence of failure. “Consumer claim activity is unavailable” is more accurate than “consumers have stopped.” Assign an owner to obtain a bounded sample from available logs or operational records.

Avoid changing worker count, timeout settings and retry behaviour together. An improved chart would not reveal which change mattered, while a new dependency problem could go unnoticed. Where containment permits, change one variable and retain the same measurement boundary for comparison.

7. Recover without losing task history

Diagnosis does not require leaving harmful activity running. Use an authorised incident procedure to contain repeated failures or dependency overload. Preserve task identifiers, failure reasons and attempt history before replaying or removing work.

Each recovery choice has a trade-off. Pausing intake shifts waiting upstream. Isolating repeatedly failing tasks can unblock healthy work, but isolated tasks still need an owner and a disposition. Restoring consumers can release enough backlog to expose a previously hidden dependency constraint.

Before replaying tasks with uncertain outcomes, reconcile whether the external side effect already occurred. A timeout does not prove nothing happened. Replay should follow the integration’s duplicate-prevention and authorisation design.

Additional capacity becomes a defensible candidate when unique demand persistently exceeds successful throughput, consumers function normally, dependencies remain healthy and repeated failures do not explain the gap. That supports evaluating capacity; it does not establish that a particular scaling change will be safe or sufficient.

8. Leave a testable checkpoint for the next operator

A handover should preserve the measurement boundary, working explanation and next test—not just a screenshot. The following filled record is hypothetical and can be reused by replacing its values.

  • Scope: integration route A; ready, in-flight and delayed-retry entries included.
  • Window: ten minutes; depth 300 to 900; unique arrivals 180 per minute; total entry arrivals 180 per minute; successful completions 120 per minute.
  • Health: oldest unresolved age 3 to 6 minutes; attempt p95 2 seconds; dependency timing and errors unchanged.
  • Working diagnosis: demand burst; no retry amplification observed.
  • Owner and next review: on-duty integration operator; review after the next ten-minute window.
  • Expected observation: if arrivals fall below 120 per minute and completions remain near 120, depth should decline.
  • Challenge check: inspect the oldest tasks to verify that progress is not restricted to newer work.

If the expected observation does not occur, revise the explanation before repeating the intervention. The useful outcome is an account another operator can test: what is accumulating, where progress stops and which evidence will determine the next action.

Integration reliabilityQueue monitoringIncident diagnosisRetry management

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications