Build an integration outage runbook with clear decision authority
Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.
The queue is growing, requests are failing, and three people are discussing whether to pause processing. One wants more retries, another wants a fallback, and nobody knows who can approve either change. The immediate problem is not a shortage of technical ideas. It is missing decision authority.
An integration outage response runbook should name who can declare an incident, approve each containment action, gather evidence, contact dependencies and sign off recovery. Separate the person who makes a decision from the person permitted to execute it. Give each action a boundary, a verification step and a route for escalation when the owner is unavailable.
The filled checklist below shows how those pieces fit together. It is a recommended operating design, not a description of controls available in any particular product. Before adopting it, confirm that each proposed action is documented and available in your environment.
1. Assign authority before choosing incident actions
Start with decisions, not job titles. A senior manager may own customer impact without having permission to change integration settings. An engineer may have technical access without authority to suspend a business process.
Assign these responsibilities and a backup for each:
- Incident commander: declares the incident, sets priorities, approves containment within agreed boundaries and keeps decisions moving.
- Technical lead: assesses evidence, recommends technical actions and verifies their effects.
- Authorised operator: executes approved changes using existing, documented permissions.
- Business owner: accepts operational trade-offs such as delayed fulfilment or a restricted fallback.
- Evidence and communications owner: maintains the timeline, coordinates updates and routes dependency questions.
- Recovery approver: confirms that technical recovery and business reconciliation are sufficient to resume normal work.
A small team can combine roles, but should record the combination explicitly. For a high-impact change, seek a second qualified reviewer where feasible. Do not make an unavailable reviewer an indefinite blocker: specify a backup with equivalent authority and define the narrower actions allowed while approval is pending.
Technical access remains a separate constraint. Appointment as incident commander does not confer access to a system. If nobody present has the necessary permission, escalate rather than sharing credentials or improvising an access workaround.
2. Establish a containment envelope
Pre-agree which decisions the incident commander can make alone, which require business approval and which are outside the runbook. Otherwise every proposed action becomes a fresh negotiation.
A useful containment envelope distinguishes three categories:
- Pre-authorised, bounded actions: a named role may approve a documented action within an agreed scope, with an operator verifying the result.
- Joint-approval actions: technical and business owners must agree because the change affects customer commitments, data handling or a fallback process.
- Unplanned actions: no standing approval exists. The team must assess the proposal, obtain the appropriate authority and document the execution and rollback plan.
Pausing processing, changing retries and activating a fallback belong on separate lines. They have different consequences. A pause may prevent further uncertain outcomes while increasing delay. Reducing retry pressure may slow legitimate work. A manual fallback may keep selected work moving but introduce reconciliation effort.
Define what remains prohibited during an outage. Examples include deleting queued work to make a dashboard look healthy, replaying records with unknown outcomes, changing undocumented settings or bypassing normal access controls. Urgency should narrow the scope of a change, not remove its owner.
3. Filled fictional outage checklist: detection to containment
Fictional worked example: All names, times, counts, thresholds and operating details in this example are hypothetical, not customer results or recommended universal targets. A CRM-to-order-processing integration has a growing backlog. Some requests return HTTP 429 responses, and some earlier requests have uncertain outcomes. The team has verified a documented way to pause this integration's dispatch while preserving accepted work.
HTTP 429 indicates rate limiting; a response may include a Retry-After header. Providers differ in how they apply limits, so the response alone does not establish a safe retry configuration. See MDN's explanation of HTTP 429.
Incident record: INT-EXAMPLE-07
- Detection — completed at 09:10. Priya, support lead, records backlog growth from 240 to 680 items over 15 minutes. Oldest pending work is 28 minutes old. She attaches the observation times and a redacted error sample. She reports the symptom without declaring a root cause.
- Declaration — completed at 09:14. Morgan, incident commander, declares an incident because delayed order processing and uncertain outcomes require coordination. Alex is named deputy. The next internal update is due at 09:30, even if the cause remains unknown.
- Role assignment — completed at 09:15. Sam is technical lead; Lee is authorised operator; Jordan owns the business process; Priya maintains evidence and communications. Morgan will coordinate recovery approval but cannot substitute for Jordan's business sign-off.
- Initial evidence — completed at 09:18. Sam records queue depth, oldest-item age, response types, recent relevant changes and the last confirmed successful processing time. Records with uncertain outcomes are listed separately from confirmed failures. Evidence stays in an access-controlled location; broad updates contain no customer payloads.
- Containment approval — recorded at 09:20. Morgan authorises Lee to pause dispatch for the affected integration only, using its documented procedure. The approval does not cover stopping unrelated integrations, deleting work or altering retry settings. Sam confirms how accepted work will remain accounted for.
- Execution and verification — completed at 09:23. Lee records the pause and its scope. Sam verifies that new dispatch attempts have stopped, checks for in-flight requests and confirms that accepted pending work remains accounted for. Intake remains open under Jordan's approval, with its resulting backlog reviewed at the next checkpoint.
- Retry decision — recorded at 09:25. Sam proposes no retry-setting change yet. The team has not verified the provider's applicable guidance or the current configuration's behaviour. Morgan records that dispatch remains paused; any retry change requires a documented proposal, authorised execution and a review time.
This checklist makes a consequential distinction: the team has approved containment, not diagnosed the dependency or approved recovery. A growing queue can be the accepted short-term cost of stopping additional uncertain transactions.
4. Log decisions so the next person can reconstruct them
A chat message saying “go ahead” is not enough. A useful decision entry connects evidence, authority and action. Keep the record short enough to update during the incident, but precise enough to distinguish a proposal from an approval and an approval from a completed change.
Filled hypothetical decision entry:
- Time and decision: 09:20 — pause affected dispatch.
- Evidence: rising backlog, repeated rate-limit responses and unresolved outcomes from earlier attempts.
- Approver: Morgan, under the pre-agreed containment envelope.
- Executor: Lee, using the documented integration-specific procedure.
- Expected effect: stop new outbound attempts; pending work may continue accumulating.
- Scope and exclusions: affected integration only; no deletion, replay, fallback or retry-setting change.
- Verification: Sam checks dispatch activity, in-flight work and queue accounting.
- Review point: 09:35, or earlier if intake cannot be accounted for.
- Reversal condition: if the pause does not behave as documented, keep recovery unapproved and escalate to the technical owner before another change.
Record rejected options when they could otherwise reappear. For example: “Bulk replay rejected because destination outcomes remain unknown.” That gives the next shift a reason, not merely an instruction.
5. Escalate dependencies and bound the fallback
Dependency escalation should be an assigned task with a specific question. Sending “the integration is down” to several contacts creates activity without necessarily producing a decision.
Continuation of the hypothetical checklist:
- Dependency escalation — completed at 09:28. Priya opens a support request through the established channel. It includes the affected period, response samples, redacted request references, observed impact and containment already performed. She asks whether the dependency can confirm request outcomes and provide applicable retry guidance.
- Escalation checkpoint — set for 09:45. Priya owns follow-up. If no useful response arrives, she uses the existing escalation route and reports the unresolved questions to Morgan. Lack of a reply does not authorise increased retry pressure.
- Fallback assessment — completed at 09:32. Jordan proposes manually handling confirmed unprocessed urgent orders. Sam identifies the risk of duplicating orders whose destination outcome is unknown.
- Fallback approval — recorded at 09:36. Jordan and Morgan approve the existing manual process only for work confirmed not to have reached the destination. Each selected item receives a reconciliation entry, a named operator and a recorded disposition. Uncertain items remain excluded.
- Fallback stop condition: Jordan must suspend the manual route if staff cannot establish eligibility or maintain the reconciliation record. The fallback is not permission to invent a new transfer method.
For a communications fallback, separately verify recipient permission, opt-outs and channel requirements. On WhatsApp, approved templates are used to initiate Platform conversations and outside the 24-hour customer-service window, which opens or resets with a user message. Template approval does not guarantee delivery. Check the current official messaging policy before adopting that route; an outage does not waive those requirements.
6. Make handover an explicit transfer of authority
A handover is incomplete until the incoming owner acknowledges responsibility. Posting a summary and leaving can create a period in which everyone assumes somebody else is deciding.
Filled hypothetical handover, 11:00: Morgan transfers incident command to Alex, who acknowledges acceptance at 11:03. Dispatch remains paused. Manual handling is limited to confirmed unprocessed urgent orders. Retry changes and bulk replay remain unapproved.
- Current impact: pending work is delayed; uncertain outcomes remain under investigation.
- Evidence location: incident record INT-EXAMPLE-07 contains response samples, queue observations and the manual reconciliation list.
- Outstanding dependency question: can the provider confirm the disposition of the uncertain requests?
- Next commitments: Priya follows up with support at 11:15; Sam presents a recovery proposal at 11:30.
- Authority transferred: Alex inherits the existing containment envelope, not permission for broader changes.
- Still required: Jordan's approval for expanded fallback scope and joint technical/business recovery sign-off.
If the incoming commander cannot accept, the outgoing commander must use the named backup route. The record should show who retains authority in the meantime, rather than leaving ownership implicit.
7. Sign off recovery separately from technical improvement
A successful test request is evidence of improvement, not proof that the backlog is safe to replay. Recovery must address both new processing and the disposition of work accumulated during the incident.
Final hypothetical checklist entries:
- Technical recovery proposal — 11:30. Sam provides fresh observations, applicable dependency guidance, the status of uncertain records and a documented restart procedure. Unresolved records remain excluded from replay.
- Limited restart approval — 11:40. Alex approves a bounded restart proposed by Sam and executed by Lee. The observation period is 15 minutes. Stop conditions include renewed errors, unexpected duplicates, unexplained queue movement or inability to verify outcomes.
- Backlog release approval — 12:00. Sam confirms that the limited restart met its checks. Jordan confirms that manually handled items are identified and excluded from duplicate processing. Alex authorises only the reconciled backlog to proceed.
- Recovery sign-off — 12:30. Sam signs off observed technical behaviour; Jordan signs off business reconciliation. Alex closes active coordination after assigning owners and review dates to the remaining exceptions. Closure does not label unresolved records as successfully processed.
If the restart fails, return to the last verified containment state using the documented procedure and authorised operator. Preserve evidence from the attempt. Reassess the failed condition before proposing another restart; repeating the same test without new evidence consumes attention and can add uncertain work.
8. Test the authority map before the next outage
Run a short tabletop exercise using one affected integration. Ask someone other than the author to identify who may pause processing, who may change retries and who may activate the fallback. Require them to locate the relevant procedure and backup approver, not merely recall a name.
Measure coordination directly: time to identify an authorised decision-maker, decisions missing an approver or review point, handovers lacking acceptance, and records still unresolved at recovery sign-off. These measures reveal runbook gaps without mistaking a shorter incident for a safer response.
Your next action is to fill the checklist for one real integration and rehearse a scenario in which its primary owner is unavailable. The exercise passes when the team can identify authorised containment, preserve evidence, transfer ownership and explain the recovery gate without improvising decision rights.
Put the next conversation to work.
Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.
Explore Communications