Inject Failures at Action Boundaries to Find Partial Completion
Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.
A workflow updates a record, creates an internal task, and sends a notification. The run then reports failure. That status does not tell you whether nothing happened, the record changed, or all the work completed but the final acknowledgement was lost.
To find out, inject controlled failures immediately before and after each action in an isolated test. Inspect the record, task store, and notification destination independently of the run summary. Document the surviving changes before cleaning up or retrying. A failure status is evidence about execution reporting, not proof that earlier actions were reversed.
This approach to automation partial failure testing produces a recovery map: where a run stopped, what still exists, what remains unknown, and which next action is safe. It does not require assuming that the workflow is transactional.
1. Define the action boundaries before choosing faults
Start with the actual sequence of side effects, not just the visible workflow steps. A single step may submit a request, wait for acceptance, and record a result. Those are different observation points.
For each action, distinguish these boundaries:
- Before submission: the action has not been sent to its destination.
- After the destination changes state: the record is saved or the task exists, but the workflow may not yet know that.
- After acknowledgement: the workflow has received a result and may proceed to the next action.
The middle boundary is particularly important. If the destination completes the action but its response never reaches the workflow, the run can appear unsuccessful while the side effect survives. Repeating the action may then create duplicate work.
Draw the sequence in plain language: update record, confirm update, create task, confirm task, submit notification, record acceptance. Mark where your test harness can actually interrupt it. If you can only stop between steps, say so. A stop before task creation does not prove what would happen if the task were created but its acknowledgement were lost.
2. Build an isolated fixture with observable state
Use a dedicated test environment or another demonstrably isolated setup. Do not discover the scope of a fault by disrupting shared credentials, blocking a shared service, or sending messages to real customers. Keep retries and independent follow-on workflows disabled or contained where the environment permits; otherwise, include their effects in the observation plan.
Fictional worked example: all identifiers, field values, timings, and results below are hypothetical. They illustrate a test design, not an Alactic capability or a report of testing.
- Test record:
TEST-418, with stageNewand ownerTest Operations. - Action A: change the stage to
Review pending. - Action B: create an internal task called
Review TEST-418, assigned to a dedicated test user. - Action C: submit an internal notification to a controlled test destination, referring to the record and task.
- Starting state: no task and no notification associated with this attempt.
- Attempt marker:
FI-418-A, included in test evidence and destination metadata where supported.
The notification is internal so that customer messaging requirements do not obscure the boundary test. It still needs a controlled destination: a notification received by a real operations team can create work even if it concerns a test record.
Before every case, capture the baseline and confirm that the previous case is no longer active. Give each attempt a distinct marker. Where the system cannot store a marker, maintain an external mapping of the run identifier, record identifier, created objects, and timestamps. Do not rely on a task title alone to identify duplicates.
3. Inject a failure without changing the question
Choose the smallest intervention that tests the boundary. A controlled test rejection before submission answers whether earlier changes survive when the next action cannot start. A response interruption after a confirmed destination write answers whether execution reporting can lose track of completed work.
Suitable mechanisms depend on the environment. A test harness might reject one selected request, pause execution at a checkpoint, or simulate loss of a response. These are recommended testing mechanisms, not claims that a particular product provides fault-injection controls. Confirm that the intended control is available before designing the case around it.
Use three rules:
- One fault per case. Do not combine a task rejection with a notification outage. You would lose the ability to attribute the residual state.
- Verify the injection point. A timeout alone does not prove whether it happened before or after the destination accepted the action.
- Separate forced stops from realistic failures. Stopping a run after an acknowledgement maps a boundary, but it does not reproduce every possible service failure.
A deliberately invalid task payload can test validation rejection, but it may not test task persistence. Likewise, an authorization failure may happen before any write. Name the condition precisely rather than treating all failures as equivalent.
4. Use a filled failure-injection matrix
The following matrix is presented as case entries so each result includes its cleanup decision. Every case starts from the hypothetical baseline above. The surviving states are illustrative observations that a tester would need to verify, not universal expectations.
Case A: Stop before the record update
- Induced failure: a harness rejects execution before Action A is submitted.
- Observed residual state: stage remains
New; no task; no notification submission. - Evidence: direct record read, task lookup, and controlled destination check match the baseline.
- Cleanup and recovery decision: no business-state cleanup. Remove the injected fault before a fresh attempt.
Case B: Lose the record-update acknowledgement
- Induced failure: the test destination saves Action A, but the harness withholds its successful response and the run stops.
- Observed residual state: stage is
Review pending; no task; no notification. - Evidence: direct record read shows the changed stage; execution evidence shows no Action B submission.
- Cleanup and recovery decision: restore
Newafter evidence capture to reset the fixture. For recovery testing, retain the update and test continuation without repeating unrelated record changes.
Case C: Reject task creation before it writes
- Induced failure: Action A succeeds; the test harness rejects Action B before task persistence.
- Observed residual state: stage is
Review pending; no task; no notification. - Evidence: record inspection confirms the update; task lookup and injection evidence confirm no task creation.
- Cleanup and recovery decision: retain the valid update for a recovery rehearsal. Create the missing task only after confirming it is absent; proceed to notification after task creation is confirmed.
Case D: Lose the task-creation acknowledgement
- Induced failure: Action B creates task
TASK-418-A, but its response does not reach the workflow. - Observed residual state: stage is
Review pending; one open task exists; no notification submission. - Evidence: task inspection confirms its identifier, record association, assignee, and attempt marker.
- Cleanup and recovery decision: preserve the existing task during recovery. Reconcile its identifier before notification; do not create another task merely because the run lacks a successful result.
Case E: Reject notification submission
- Induced failure: Actions A and B succeed; Action C is blocked before submission.
- Observed residual state: updated record and one open task survive; no notification is accepted by the controlled destination.
- Evidence: record and task inspection, plus a captured pre-submission rejection.
- Cleanup and recovery decision: retain the record and task. Test notification-only recovery if supported; otherwise use an authorized manual step rather than replaying the entire sequence blindly.
Case F: Lose notification acceptance acknowledgement
- Induced failure: the controlled destination accepts Action C, but the response is withheld.
- Observed residual state: updated record, one task, and a notification acceptance record survive. Recipient receipt is not established by acceptance alone.
- Evidence: destination acceptance identifier matches the attempt marker; the workflow reports failure or an unresolved outcome.
- Cleanup and recovery decision: reconcile destination status before resending. Preserve evidence of acceptance. Treat any notification already received as a side effect that restoring the record cannot undo.
Cases B and C leave the same visible business state, but for different reasons. Cases D and E both leave a task, yet only one has a missing task acknowledgement. Those distinctions matter when deciding whether an automatic retry would repeat completed work.
5. Inspect residual state independently of the run summary
After inducing the fault, stop further test activity and inspect each destination. A log entry saying that an action started is not proof it completed. A missing success entry is not proof it did not.
Capture the record's relevant fields, the identifiers and status of matching tasks, notification submission or acceptance evidence, and any pending retries. Record both what you found and where you looked. An empty search result is weaker evidence if the search excludes completed tasks or depends on a title that could have changed.
Inspect immediately and again after a predefined observation interval suited to the test environment. For this hypothetical fixture, the tester might choose a second inspection after two minutes. That interval is not a general recommendation or proof that later effects are impossible. Its purpose is to expose delayed activity within a declared observation window.
Use an explicit unknown state when evidence is insufficient. For example: notification acceptance unknown; no destination lookup available. Do not replace uncertainty with “not sent.” A boundary is not fully characterized until the team either obtains evidence or documents the visibility gap and a conservative recovery rule.
6. Decide what to retain, reverse, or reconcile
Cleanup for repeatable testing and recovery of useful work are different decisions. Resetting every fixture to its baseline makes cases comparable. Reversing every surviving action in a real business process may be unnecessary or harmful.
- Retain a valid change when the intended business state still applies and the remaining work can be completed safely.
- Reverse a test change when restoring the fixture, provided no other activity has legitimately changed that state since the test began.
- Reconcile when an action may already have completed but its result is missing from the workflow's record.
- Escalate for review when evidence is insufficient or the next action could duplicate work.
A reversal is another action that can fail. Verify its result and record what it did not undo. Closing a test task does not retract a notification, and restoring a stage does not remove work someone already began.
For retries, distinguish an attempt marker from a business-operation identity. Every attempt needs separate evidence, while repeated attempts at the same intended operation need a way to recognize existing work. If the design supports duplicate prevention, test it explicitly. Do not assume a run identifier or a repeated field assignment protects every downstream action.
7. Turn the findings into a recovery rehearsal
Choose the most consequential residual state from the matrix and rehearse its recovery in isolation. In the hypothetical example, Case D is a useful candidate: a task exists, but the workflow does not know its identifier. The rehearsal should establish whether the team can find that task, associate it with the failed attempt, and finish the notification step without creating another task.
Set observable acceptance criteria before running the rehearsal: the intended record stage remains correct, exactly one task exists for the intended operation, the notification outcome is documented separately from submission, and no unexplained retry remains pending.
As a concrete next action, select one multi-action workflow and complete the six boundary cases above using isolated records. Report the number of cases with confirmed residual state, the number still unknown, and any duplicate objects created during recovery. Assign an owner to every unknown outcome before enabling unattended retries for that path. The useful result is not a green run summary; it is evidence that a stopped run can be understood and continued without guessing what already happened.
Put the next conversation to work.
Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.
Explore Communications