Back to all articlesAPI integration planning

Decide whether sandbox gaps should block integration approval

Alactic Editorial10 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

The sandbox review ends with three reassuring phrases: most tests passed, the remaining cases are sandbox limitations, and the team will monitor production. None tells an approver whether an unresolved case could expose another customer's records, repeat a business action, or merely delay an internal status update.

Use this rule for API sandbox testing exit criteria: block an unknown when its plausible consequence is unacceptable and no verified boundary contains it; narrow scope when the affected path can genuinely be removed; approve an exception only when the remaining consequence is understood, bounded, and owned. A sandbox limitation explains missing evidence. It does not, by itself, justify acceptance.

The decision log below turns that rule into a practical review. It separates the person who must obtain evidence from the person authorised to accept the business consequence. It also makes approval expire or reopen when its supporting assumptions change.

1. Define exactly what the approval covers

Before debating incomplete tests, write a short release boundary. Identify the business workflow, permitted users, data touched, enabled actions, and excluded paths. An approval for a read-only internal view is not an approval for automated updates to customer records.

Describe the boundary in operational terms, not just feature names. For example, say whether a person reviews each proposed change before execution, whether external recipients can be contacted, and whether the integration can change ownership of a record. These distinctions determine the consequence of a missing test.

Record the integration build and configuration under review, plus the evidence location. If the team changes permissions, retry behaviour, or enabled actions after testing, the old approval may no longer describe the proposed release.

Recommended exit statement: Approval covers only the listed workflow and configuration. Every unresolved requirement has a disposition, an evidence owner, an accountable approver, and a revisit condition. Removed scope has a verified exclusion. A blocking item prevents the affected scope from being approved.

This avoids a common failure: a narrow test result being treated as broad permission to launch adjacent workflows.

2. Separate an environment limit from an integration assumption

For each gap, write two separate sentences: what could not be observed, and what the team currently assumes. They are not interchangeable.

  • Environment limit: The sandbox did not generate the provider response needed to exercise the retry path.
  • Assumption: The integration will pause and resume without repeating a business action.
  • Evidence needed: A controlled test of the integration's response handling, plus confirmation of any provider behaviour that the design depends on.

A simulated response can establish how your code behaves when it receives that response. It cannot establish that a provider will produce the same response, headers, timing, or event sequence in every relevant situation.

Keep evidence claims precise. For example, HTTP 429 indicates rate limiting, and a Retry-After header may indicate when to retry; providers differ in how they apply limits. Those facts do not prove a particular integration's recovery behaviour. See the MDN explanation of HTTP 429 when documenting this distinction.

Label evidence as directly observed, simulated, documented, or unknown. This is more useful than a single confidence score that conceals why the team feels confident.

3. Classify gaps by consequence, not test difficulty

A test that is expensive to reproduce is not necessarily safe to waive. Equally, an untested cosmetic state need not block every business workflow. Assess each gap against four questions:

  1. What can go wrong? Name the business action or data exposure, not merely the component failure.
  2. Who or what is affected? Include people who are not operating the integration, such as customers whose records it processes.
  3. Can the effect be reversed? Correcting an internal display is different from retracting information already disclosed.
  4. What verified boundary limits exposure? Identify an enforced permission, disabled path, review step, or other demonstrated constraint.

Then choose a disposition:

  • Block: The gap concerns a necessary property of the proposed scope, and the plausible consequence is not acceptably contained.
  • Narrow scope: The uncertain path is separable, and the team can demonstrate that it is unavailable in the approved release.
  • Accept a documented exception: The workflow remains in scope, but its known limitation has a bounded consequence, a workable fallback, and explicit acceptance.

Do not average severe unknowns against successful tests. Many passed formatting checks do not compensate for missing evidence that users cannot access one another's records. OWASP recommends authorising access to each requested object; possession of an object identifier is not sufficient. Its object-level authorization guidance supports treating this as a distinct requirement, not a general confidence question.

4. Filled hypothetical sandbox exit decision log

Fictional worked example: All roles, workflows, thresholds, and observations below are hypothetical. They illustrate a recommended decision process, not an existing product control or customer result.

A team proposes an internal integration that reads customer records, shows request status, creates follow-up tasks after staff confirmation, and optionally synchronises bulk record updates. The sandbox review has unresolved cases. The release candidate is labelled RC-A; its configuration and test evidence are retained with the decision log.

Entry A: Cross-account record access — block

  • Requirement: A user must not retrieve a customer record outside their authorised account.
  • Missing evidence: The sandbox supplied only one account context. The team has not tested a valid record identifier from another account.
  • Consequence: An unauthorised user could see another account's customer information. Disclosure would not be reversed by correcting the permission later.
  • Available evidence: Same-account reads passed. That does not establish cross-account denial.
  • Decision: Block approval of record-reading scope. Do not treat successful sign-in as proof of object-level permission checks.
  • Evidence owner: Integration engineering lead. Obtain an authorised test arrangement with separate account contexts; record expected denial, actual response, and absence of returned record data.
  • Accountable approver: Data owner, with security review of the evidence.
  • Revisit condition: Reconsider after cross-account negative tests pass for the relevant access paths. Repeat review if permission mapping or account-routing logic changes.

Entry B: Bulk synchronisation under rate limiting — narrow scope

  • Requirement: Bulk synchronisation must handle throttling without losing updates or repeating completed business actions.
  • Missing evidence: The sandbox did not reproduce the provider's throttling sequence. Local simulations exercised selected responses but did not verify provider-specific behaviour.
  • Consequence: Bulk work could become incomplete or duplicated, requiring reconciliation across many records.
  • Decision: Remove bulk synchronisation from this release. Retain only individually reviewed actions, subject to their separate acceptance tests.
  • Verified boundary: The candidate configuration disables the bulk trigger. A negative test confirms that the normal user path cannot submit bulk work; the review also checks scheduled and alternate entry paths.
  • Evidence owner: Integration engineer. Retain exclusion-test results and prepare a controlled test plan for retry, partial completion, and reconciliation.
  • Accountable approver: Business operations owner accepts the reduced workflow, not the untested bulk capability.
  • Revisit condition: Any proposal to enable bulk work requires a new decision supported by provider-specific evidence and recovery tests.

Entry C: Delayed internal status display — accept a limitation

  • Requirement: Staff should be able to determine whether a request needs follow-up.
  • Missing evidence: Exact status-update latency under production load is unverified.
  • Known boundary: The proposed integration uses the status only for an internal display. Tests confirm that it does not trigger external communication or another write action.
  • Consequence: Staff may spend extra time checking a request. The team has not accepted an unknown completion time as a delivery commitment.
  • Decision: Accept the display limitation for a hypothetical pilot of five trained staff, provided the display identifies its last refresh and staff use the authoritative record before repeating an action.
  • Evidence owner: Test lead verifies the refresh indicator and the manual checking procedure before approval.
  • Accountable approver: Operations manager accepts the additional checking workload.
  • Revisit condition: Review after five working days, before expanding access, or immediately if staff cannot determine the authoritative state. The hypothetical exception expires after ten working days unless explicitly renewed.

Entry D: Ambiguous response after a confirmed write — block the write path

  • Requirement: Staff must be able to resolve an uncertain result without blindly repeating a task-creation request.
  • Missing evidence: The team has not demonstrated what happens when a write succeeds but its response is lost.
  • Consequence: A repeated request could create duplicate follow-up work, while avoiding a repeat could leave work unconfirmed.
  • Decision: Block task creation until a tested reconciliation procedure distinguishes completed, failed, and unresolved requests. A read-only release remains a separate option after Entry A is cleared.
  • Evidence owner: Application engineer, supported by the operations lead who must validate the checking procedure.
  • Accountable approver: Workflow owner.
  • Revisit condition: Re-run the ambiguous-result test and demonstrate a usable resolution path. Reopen the decision if request tracking or write behaviour changes.

Overall decision: Do not approve the full proposed integration. Bulk synchronisation is excluded; task creation remains blocked; record reading remains blocked pending authorization evidence. The status-display exception does not override either blocker. A narrower candidate can be submitted once its prerequisites are met.

5. Make evidence ownership different from risk acceptance

The evidence owner answers, “Who will obtain the missing proof?” The accountable approver answers, “Who can accept the remaining business consequence?” Naming only the developer leaves the second question unresolved. Naming only a sponsor leaves nobody responsible for closing the evidence gap.

Each owner needs a concrete deliverable. “Investigate retries” is not enough. “Record the outcome when a successful write loses its response, and demonstrate how staff identify that outcome” is reviewable.

Set a review date and a disposition if evidence is still absent. Missing evidence should not quietly become acceptance because a calendar date arrived. The available choices remain block, remove the path, or seek explicit acceptance of a sufficiently bounded limitation.

Where a fallback depends on people, verify their access, instructions, and capacity. A manual check is not a control if the assigned staff cannot see the authoritative record or cannot realistically perform it.

6. Test the boundary that makes the decision defensible

Scope reduction trades capability for a smaller evidence burden. It works only if the removed capability is actually unavailable. Hiding a button is inadequate when another route can still invoke the same action. Review the applicable user, scheduled, administrative, and background paths.

An exception makes a different trade: the workflow stays available, but the business accepts a defined limitation. That acceptance needs evidence about containment even when evidence about ideal behaviour is incomplete.

For an uncertain write, monitoring alone is weak containment. Seeing a duplicate later does not prevent it. A better approval basis may require removing the write path until a usable reconciliation method is demonstrated. Similarly, permission checks cannot be replaced by a promise to review access logs after release.

If a boundary test fails, return the item to blocked status or remove more scope. Preserve the failed result and the revised decision so the next review does not restart from an optimistic summary.

7. End the review with a measurable decision, not conditional enthusiasm

Before closing the meeting, count unresolved requirements and check that every one has a disposition, named owners, evidence location, and revisit condition. Separately count blocking items within the proposed approval boundary. That count must be zero before that boundary is approved.

Record what would invalidate each exception: wider access, new write permissions, a changed dependency, an expired review date, or a failed containment check. Reopening a decision means the affected scope needs fresh approval; it is not merely a note for a future meeting.

Next action: Take the current unresolved test list and complete these fields for every item before the next approval review: requirement, missing evidence, business consequence, verified boundary, disposition, evidence owner, accountable approver, and revisit trigger. Then write one sentence stating exactly what can proceed and what cannot. If that sentence depends on an unverified safeguard, the decision is not ready.

API integrationSandbox testingRelease approvalEvidence management

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications