Define Expected Results Before Running an Automation Test
Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.
A record changed, a task appeared, and the run displayed a success message. One tester calls that a pass. Another finds that the task belongs to the wrong person. A third notices that an unrelated record also changed. The disagreement started before execution: nobody had agreed what success required.
Define expected results as observable assertions before running the test. For each input, specify the starting state, exact permitted changes, actions the test identity is authorised to perform, changes that must not occur, and evidence needed for a verdict. A pass requires all mandatory assertions to hold—not merely visible activity.
The expected-results ledger below makes those decisions explicit. It is a recommended testing practice, not a claim that any particular product exposes every evidence source described. Check test visibility in the current workspace and plan before relying on it.
1. Replace “the automation worked” with a testable contract
A useful test oracle is an agreed rule for deciding whether an observed result is correct. For a business workflow, that rule usually covers more than the main output.
“Create a follow-up task” leaves important questions unanswered. Which record should the task reference? Who owns it? What happens if the event arrives again? May the workflow change the record owner? Does missing permission stop the action, produce a partial result, or send it for human handling?
Write assertions at the level where those differences become visible:
- Target: identify the exact record or object that may change.
- Value: specify the resulting field value, not just that an update occurred.
- Quantity: define how many tasks or other objects may be created.
- Authority: name the identity performing the action and its required permission.
- Boundary: list relevant objects, fields, and external actions that must remain untouched.
- Observation period: define when evidence will be collected and how an unfinished run will be classified.
These assertions form a contract for the test. Agree it before execution so that an unexpected result cannot quietly become the new definition of success.
2. Establish the starting state and available evidence
A final value means little without a baseline. If a task already existed before the run, finding it afterward does not prove the automation created it. If a record already had the expected status, an unchanged screen cannot prove that the update step ran.
Capture a baseline using stable identifiers wherever available. Include the target record, relevant linked objects, fields allowed to change, and selected fields that must not change. Record the test identity's effective permissions, including whether it can act on that specific object.
Next, confirm what evidence the workspace actually exposes. Useful evidence may include before-and-after record views, task searches, activity history, execution records, or permission-denial details. Availability and detail can differ by workspace and plan. Do not write a test that depends on an audit trail until you have confirmed that it exists and is accessible.
Different evidence answers different questions. A run status may show that processing finished; it does not establish that the correct record changed. A task search may prove that a task exists; it may not establish which run created it. Use complementary evidence where necessary.
If an essential assertion cannot be observed, mark the test blocked or inconclusive. Missing evidence is not a pass, but it is also not proof that the underlying action failed.
3. Filled hypothetical expected-results ledger
Fictional worked example: Every identifier, value, timing choice, and observation in this ledger is hypothetical. This example describes a proposed CRM test design, not an existing Alactic feature or a customer result.
The workflow under test handles a “qualified” event. Its intended result is to update one lead and create one internal follow-up task. It must not contact the lead, change ownership, or modify unrelated records. The proposed design treats a repeated event as a duplicate rather than a request for another task.
Shared test fixture
- Target record: lead
L-104, statusNew, ownerPriya, qualification note empty, no task linked to eventEV-208. - Control record: lead
L-105, statusNew, ownerSam. Relevant fields and linked tasks are captured before execution. - Primary identity:
Test Operator A, authorised to update the target's status and qualification note and create its internal task, but not to change ownership or send messages. - Evidence window: inspect after processing reports completion, with a hypothetical observation limit of two minutes. If processing remains unresolved, use an inconclusive verdict rather than guessing.
- Evidence package: baseline capture, input identifier, effective permissions, final record views, linked-task search, and execution or activity details where available.
Ledger entry A: authorised qualifying event
- Starting state: shared fixture unchanged; no prior processing of
EV-208. - Input event:
EV-208targetsL-104with qualification noteRequested a product discussion. - Permitted changes: status becomes
Qualified; qualification note matches the input; exactly one task is created with titleReview qualified lead L-104, linked toL-104and assigned to Priya. - Forbidden changes: no owner change, no outbound communication, no additional task, and no relevant change to
L-105. - Evidence to inspect: target fields, task count and attributes, control-record comparison, and available evidence of outbound actions. Correlate observations with
EV-208where supported. - Hypothetical observation: target values match; one correctly linked and assigned task exists; owner and control record are unchanged; the available action record shows no outbound attempt.
- Final verdict: pass for the declared assertions and observation period.
Ledger entry B: replay of the same event
- Starting state: entry A's final state, including its single task.
- Input event: replay
EV-208without changing its identity or payload. - Permitted changes: none to business records or tasks. A processing entry may record that the input was already handled.
- Forbidden changes: no second task, repeated note content, owner change, or outbound action.
- Evidence to inspect: linked-task count and identifiers before and after replay, target-field comparison, and available processing details.
- Hypothetical observation: a second task appears, although both executions report success.
- Final verdict: fail. The duplicate violates the agreed result even though processing completed.
Ledger entry C: identity lacks task-creation permission
- Starting state: a fresh isolated copy of the original fixture.
Test Operator Bcan view the target but cannot perform the required updates or create its task. - Input event: a new qualifying event targeting that fixture.
- Permitted changes: no business changes. The proposed failure path records the denial and makes it available to the designated human reviewer.
- Forbidden changes: no status update, task creation, outbound action, or use of another identity's privileges.
- Evidence to inspect: effective permissions, unchanged target and task list, and the available denial or review evidence.
- Hypothetical observation: business data remains unchanged, but the reviewer cannot access evidence that distinguishes denial from an unprocessed event.
- Final verdict: inconclusive overall. The no-change assertions hold, but the required failure-path evidence is missing.
4. Test authorisation separately from output correctness
A correct-looking output can still result from an unauthorised action. Do not treat the ability to trigger a workflow as permission for everything it might do.
For each consequential action, identify the acting identity, target object, requested operation, and permission required. Test both an allowed case and a denied case. Possession of a record identifier alone should not establish authority to modify that record; this distinction is reflected in OWASP's object-level authorisation guidance. That guidance is a design reference, not evidence about a particular product.
Decide the expected failure behaviour in advance. The fictional ledger requires no business changes when the necessary permissions are absent. Another workflow might intentionally allow an earlier independent step to remain completed. Neither expectation should be inferred after the run.
If partial completion is permitted, list exactly which changes may remain and how a human can identify unfinished work. Otherwise, testers may disagree over whether a changed record without its required task is acceptable progress or a failed result.
5. Make prohibited side effects inspectable
“Nothing else changes” is too broad to verify. Narrow it to meaningful risk boundaries: ownership, unrelated records, duplicate tasks, recipient contact, deletion, and access to other objects.
Choose evidence that matches each boundary. Comparing the target record cannot reveal a duplicate task elsewhere. Looking at a recipient conversation cannot necessarily establish that no outbound attempt occurred. A control record helps detect some unintended updates, but it does not prove that every other record remained untouched.
State that limitation in the test scope. The fictional ledger checks its named control record and available action evidence; it does not claim exhaustive coverage of the entire workspace.
Also distinguish business changes from incidental metadata. If a system-managed processing timestamp is allowed to change, list it as an explicit exception. Do not exclude fields merely because an earlier run changed them unexpectedly. An exception needs a reason consistent with the workflow's intended behaviour.
6. Use verdict rules that preserve uncertainty
Agree on verdicts before anyone inspects the outputs:
- Pass: every mandatory assertion is supported by adequate evidence within the declared scope.
- Fail: evidence contradicts at least one mandatory assertion, including a forbidden side effect.
- Inconclusive: execution occurred or may have occurred, but evidence is insufficient to decide.
- Blocked: a prerequisite prevents a valid test, such as an unavailable permission configuration or missing baseline.
A known violation should not disappear behind missing evidence. If a duplicate task is visible but outbound evidence is unavailable, the run still fails on duplication; note that the outbound assertion remains unresolved.
Keep timing separate from correctness. In the hypothetical ledger, the observation limit is a test decision, not a product performance promise. If a late task appears after an inconclusive verdict, retain the original observation and append the later evidence. Do not silently rewrite the result.
7. Recover from failed runs without destroying the evidence
When a run fails, first preserve the baseline, input, outputs, and relevant identifiers. Then contain unwanted follow-on actions where the environment permits. Avoid immediate replay: it can create more duplicates or blur which execution caused which change.
Separate diagnostic inspection from cleanup. Record any manual correction, including deleted test tasks or restored fields, so that the next tester does not mistake repaired data for the original result.
Rerun against a fresh fixture or a verified reset state. Changing the workflow and rerunning against already modified records tests a different starting condition. That can be useful for recovery testing, but it is not a clean repeat of the original test.
After a fix, repeat both the failed assertion and nearby boundaries. A change that prevents duplicate tasks might also suppress legitimate new events. Test the replay and a distinct valid event rather than checking only that the duplicate disappeared.
8. Prepare the next run with an evidence-ready checklist
Choose one workflow and complete its ledger before triggering it. Keep the first scope small enough that another tester can independently reach the same verdict.
- Identify the target records, acting identity, input, and clean starting state.
- Write exact permitted field changes and object-creation counts.
- List prohibited actions and relevant unchanged fields or objects.
- Specify the permission-denied and repeated-input outcomes.
- Map every mandatory assertion to evidence available in the current workspace and plan.
- Set the observation period and verdict rules.
- Have another tester review the ledger without seeing an execution result.
The measurable readiness check is simple: every mandatory assertion has an observable value, an evidence source, and a verdict rule. Any blank means the test needs more preparation. After execution, compare the testers' verdicts and investigate disagreements at the assertion level—not by voting on whether the automation looked active.
Put the next conversation to work.
Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.
Explore Communications