Back to all articlesIntegration reliability

Detect silent integration stalls when no errors are being reported

Alactic Editorial10 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

An integration can stop moving work while its dashboard remains green. A worker may still be running but no longer collecting jobs. A queue may accept records without releasing them. Or the monitoring feed itself may have stopped updating, leaving yesterday’s healthy status on screen.

To detect an integration that has stopped working without errors, combine five signals: whether work should exist, when meaningful progress last happened, how pending work is moving, whether the observations are fresh, and whether an authorised isolated check can complete. An old completion timestamp alone is not enough, especially during low activity.

The operating question is not simply “Has anything happened recently?” It is “Was eligible work due, and can we show that it advanced within its expected time?” The checklist below turns that question into an investigation that distinguishes a quiet business period, a stopped worker, a frozen queue, and broken telemetry.

1. Define the work that should be moving

Start with an independent reason to expect activity. Otherwise, every quiet evening can look like an outage, while a busy integration can appear healthy because unrelated activity continues.

Useful expected-work signals include an eligible source record awaiting synchronisation, a scheduled import whose start time has passed, or an accepted event that has not reached its next processing stage. Choose signals close enough to the source that they do not depend entirely on the worker being monitored.

Separate three conditions:

  • No work exists: nothing is currently eligible for processing.
  • Work exists but is not due: records are intentionally scheduled, paused, awaiting approval, or blocked by a known prerequisite.
  • Work is due: at least one eligible item should now be moving.

This distinction prevents an approval hold from becoming a false worker incident. It also prevents a scheduler failure from disappearing behind an empty queue: the schedule can say work was due even when the scheduler never created a job.

Define “done” at the same time. For a record synchronisation, it might mean that the intended destination state was confirmed. Acceptance into a queue is an earlier milestone, not proof of completion. Track intermediate stages separately so that a stall can be located rather than merely announced.

2. Measure meaningful progress, not just activity

A process heartbeat proves only that something emitted a heartbeat. Repeated polling, retries, or lease renewals may continue while business work remains stuck.

Maintain separate observations for the last worker heartbeat, last item collected, last meaningful stage transition, and last confirmed completion. A meaningful transition advances an item toward its intended outcome; writing another retry log does not qualify.

For pending work, observe more than queue depth:

  • Oldest eligible item age: how long the longest-waiting due item has waited.
  • Arrivals and departures: whether items are entering and leaving the relevant stage.
  • Item identity: whether the same items remain stuck while others pass through.
  • In-flight state: whether collected work is completing, waiting, or repeatedly being reclaimed.

A stable queue depth is ambiguous. Equal arrivals and departures can keep the count unchanged during healthy processing. Conversely, a shrinking queue can hide items being discarded or moved to an exception state. Count successful outcomes separately from removals.

Record both when an event happened and when the monitoring system observed it. A recently received update about an old completion should not make current progress look fresh. Use a consistent time basis, and investigate timestamp inconsistencies before relying on small differences.

3. Make low-volume detection conditional

Use a rule based on evidence, not a universal inactivity timeout:

Suspect a stall when work is due, meaningful progress is overdue, and fresh observations show that the due work has not advanced.

If observations are stale, classify the result as an observability problem or an unknown state. Do not silently treat missing data as either healthy operation or confirmed integration failure.

Choose the expected progress interval for each stage. It should reflect processing schedules, intentional delays, normal completion variability, and the business cost of waiting. A single global timer is unlikely to fit both immediate event processing and a scheduled batch.

Hypothetical rule: a small record-sync workflow normally advances due items within five minutes. Its team investigates when the oldest eligible item exceeds fifteen minutes and no meaningful transition appears across two fresh observations five minutes apart. These are fictional settings, not recommended defaults or measured benchmarks.

Low-volume workflows need an additional branch. If no work is due, report “idle; processing path not recently exercised,” rather than claiming full end-to-end health. Where justified, use an isolated synthetic check to reduce that uncertainty. This creates a more honest status model: progressing, intentionally idle, suspected stall, or visibility unknown.

4. Filled hypothetical stall-detection checklist

Fictional worked example: a business synchronises approved internal records from a source system to a destination workspace. The following observations, times, quantities, and thresholds are hypothetical. Each scenario is an alternative state of the same workflow, not an Alactic customer result.

The operator checks source eligibility, progress timestamps, queue movement, observation freshness, and a pre-authorised test route. The workflow is expected to process eligible records continuously during its operating window.

Scenario A: quiet business period

  • Expected work: the source query shows no eligible records; no scheduled run is overdue.
  • Progress: the last confirmed completion was seventy minutes ago.
  • Queue movement: pending and in-flight counts are both zero in two observations five minutes apart.
  • Monitoring freshness: worker and queue observations were collected within the last minute; the source query is current.
  • Isolated check: an authorised test record reaches its expected internal destination in two minutes.
  • Assessment: quiet period, with the tested route currently working. The old completion timestamp is not evidence of a stall.
  • Evidence before escalation: none currently warrants a stall escalation. Escalate if due work appears and fails to advance, or if an applicable isolated check fails. Preserve the scope of what the successful check actually covered.

Scenario B: stopped worker

  • Expected work: six approved source records are due.
  • Progress: no item has been collected for twenty-four minutes; the worker heartbeat is also twenty-four minutes old.
  • Queue movement: six eligible items remain pending; no departures occur across two fresh observations.
  • Monitoring freshness: an independent queue query and collector timestamp are current.
  • Isolated check: the test item enters the isolated route but remains uncollected.
  • Assessment: a stopped or unreachable worker is suspected. A stale heartbeat alone would not establish that diagnosis.
  • Evidence before escalation: attach due-item references, queue observation times, the missing pickup history, and an independent worker-runtime check. Route to the worker owner without waiting for an application error.

Scenario C: frozen queue or dequeue path

  • Expected work: nine eligible items are waiting, including the isolated test item.
  • Progress: worker heartbeats remain current, but the last pickup occurred twenty-eight minutes ago.
  • Queue movement: arrivals are visible; departures remain zero. The same oldest eligible item persists across observations.
  • Monitoring freshness: queue queries and worker observations are current.
  • Isolated check: the test item is accepted but does not advance.
  • Assessment: the dequeue path is suspect, not proven to be a failed queue service. A consumer pause, routing mismatch, or eligibility mistake could produce similar evidence.
  • Evidence before escalation: verify that the items are due, mapped to the intended consumer, and not intentionally paused. Include enqueue references, queue identity, pickup history, and independently observed worker state.

Scenario D: broken telemetry

  • Expected work: three records became due after the dashboard’s last update.
  • Progress: the dashboard shows no recent completion, but an authorised destination read confirms the expected records arrived.
  • Queue movement: the dashboard count is unchanged; a direct queue query shows the items have left.
  • Monitoring freshness: the dashboard collector timestamp is thirty-five minutes old.
  • Isolated check: the isolated test completes at the destination but never appears in the dashboard.
  • Assessment: telemetry failure is supported; processing is working for the checked records. Overall visibility remains degraded.
  • Evidence before escalation: provide destination confirmations, direct queue observations, the stale collector timestamp, and the missing test trace. Escalate to the monitoring owner while maintaining a temporary independent check.

5. Use synthetic checks without creating customer activity

A synthetic check is useful when natural traffic is too sparse to demonstrate progress. It should be a small, authorised operation through an explicitly isolated route, not an unannounced experiment on live customer records.

Before enabling it, document the owner, permitted environment, test identities, expected stages, cleanup method, and stop conditions. Use clearly identifiable test records and correlation identifiers. Restrict permissions to the objects and actions required.

For this detection pattern, exclude real customer messaging. Ensure the test cannot trigger external notifications, customer campaigns, billing actions, or downstream business decisions. If that isolation cannot be established, do not run the check; use read-only evidence and report the remaining uncertainty.

A passing check has limits. A dedicated test route may exercise the worker but bypass a production routing rule. A test record may not encounter the approval prerequisite affecting real records. Document what the check covers and compare it with the actual stalled stage before treating success as reassurance.

Do not keep injecting test items into an apparently frozen queue. Repeated probes can create noise and a recovery backlog. Stop at the agreed limit and retain the existing item’s trace.

6. Alert on a decision, not a blank chart

An actionable alert should contain the evidence needed to choose a next step:

  • The workflow and stage affected, with the accountable owner.
  • Why work was expected and which eligibility rule was applied.
  • The oldest due item, last meaningful progress time, and observation times.
  • Queue arrivals, successful departures, and relevant in-flight state.
  • Telemetry freshness and any independent corroboration.
  • The isolated check result, if used, including its coverage limits.
  • The suspected failure class and the safest next investigation.

Keep “progress stalled” separate from “monitoring stale.” They may involve different responders, and both can happen together. A broken dashboard does not rule out a worker failure.

Repeated observations reduce transient false alarms but delay detection. Independent checks increase confidence but add operational complexity. Set the balance according to how long the business can tolerate due work waiting, rather than making every integration page someone after the same idle interval.

7. Recover without turning a stall into duplicates

Before restarting or replaying anything, preserve the affected item references, stage transitions, queue state, and timestamps. Check for a deliberate pause or unmet prerequisite. Restarting a worker will not resolve a record that is correctly waiting for approval.

Then establish whether any work may already have completed without being acknowledged. Where destination effects are uncertain, reconcile them before replaying. Use existing duplicate-prevention controls where available, but do not assume a restart or replay is automatically safe.

A single blocked item may coexist with healthy throughput. Investigate item-level age and repeated attempts instead of declaring the entire integration recovered because newer records are completing.

Recovery needs outcome evidence: affected due items advance, expected destination state is confirmed, backlog age declines, and monitoring stays fresh. A current heartbeat or a cleared alert is insufficient. If a synthetic check passes while the original items remain stuck, the incident is not resolved for those items.

8. Put one workflow under observation tomorrow

Choose one low-volume integration and write down its eligible-work rule, meaningful completion event, expected progress interval, independent freshness check, and escalation owner. Capture two observations using the checklist before enabling a paging rule.

If an isolated check is authorised, record its boundaries and expected outcome. Otherwise, explicitly label idle periods as untested rather than healthy end to end.

Measure the result by whether the operator can answer three questions from the collected evidence: Was work due? Did it advance? Are these observations current? If any answer is unavailable, the immediate task is to close that visibility gap—not to shorten an inactivity timer.

Integration reliabilityMonitoringIncident responseOperational checklists

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications