Set stopping rules before checking campaign test results
Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.
The dashboard shows variant B ahead, so someone asks to end the test. Yesterday, A was ahead. If the team keeps checking until the result looks convincing, the stopping decision becomes part of the experiment—even if nobody documented it.
Set stopping rules before opening comparative results. Choose a fixed-horizon design with a scheduled analysis, or a sequential design whose statistical method supports the planned monitoring. In either case, separate recipient protection and operational checks from decisions that declare a performance winner.
The deliverable is a monitoring plan: when recruitment ends, how outcomes mature, what can pause activity, and which conclusions remain permissible afterward.
1. Why stopping at the first convincing lead fails
A conventional fixed-horizon analysis is intended for its specified analysis point. Repeatedly applying an ordinary significance threshold and stopping when it is crossed changes the decision procedure. The uncertainty reported by the selected snapshot no longer describes the procedure actually followed.
This is not only a small-sample problem. A large experiment can also mislead if the team searches for a favourable stopping point. Checking additional metrics, segments and attribution windows creates further opportunities to select an attractive result.
Immature outcomes create another problem. A message generating quick replies may look stronger today than one generating later purchases. Comparing them before every recipient has the planned purchase window answers a different question.
A useful stopping rule therefore specifies both the analysis time and the observations eligible for that analysis.
2. Choose the decision method before launch
Fixed horizon: Define the primary metric, assignment unit, sample target, recruitment schedule, outcome window and final analysis time. Comparative performance cannot bring the decision forward. Specify a shortfall rule so a missed target does not become an excuse to keep recruiting until significance appears.
This approach is useful when the business can wait. Its main operational advantage is clarity: monitoring protects the campaign, while one scheduled analysis supports the performance decision.
Sequential: Use a reviewed method that explicitly supports interim decisions, such as a suitable group-sequential design or always-valid approach. Document decision boundaries, eligible observations, monitoring frequency, maximum duration and any futility rule.
A continuously refreshing dashboard is not itself a sequential method. Neither is rerunning an ordinary test every morning. Futility also does not establish equivalence; it means continuing is not worthwhile under the specified rule.
If the team cannot explain the method and its permitted decisions, a fixed horizon is the clearer default. Verify any experimentation support in the actual workspace and current plan; the operating design below does not assert that a product supplies these controls.
3. Separate protective monitoring from winner selection
Keep three purposes distinct:
- Recipient and operational protection: permission handling, opt-outs, message accuracy and service capacity.
- Experiment integrity: assignment consistency, allocation, tracking completeness and unchanged eligibility.
- Performance decisions: whether the planned primary analysis supports choosing a variant.
A recipient-handling failure can justify an immediate pause without statistical significance. It cannot establish which message performs better. For WhatsApp activity, recipient permission matters and opt-outs must be honoured; use the current WhatsApp Business Messaging Policy for precise requirements.
If a guardrail such as opt-out rate also supports a statistical comparison, specify that analysis separately. An incident trigger and a statistical difference answer different questions.
Where practical, give operations staff assignment counts and health indicators without comparative primary outcomes. Accidental exposure should be logged, not used to move the endpoint.
4. A filled monitoring plan to reuse
Fictional worked plan: All dates, quantities, thresholds and results in this example are hypothetical. This is a recommended design, not an Alactic feature specification or customer result. The sample target requires separate prelaunch justification; it is not a claim of adequate statistical power.
Question and measurement
- Decision: Whether to replace reference message A with alternative B for a subsequent campaign to the same defined audience.
- Treatment: Wording only. Offer, eligibility, scheduling rules and follow-up treatment stay fixed.
- Assignment: Each eligible recipient is randomised once. Repeat activity remains attached to that assignment. Use equal allocation within each day's quota.
- Primary metric: Percentage of assigned recipients completing at least one tracked purchase during the seven full days following assignment.
- Denominator: All randomised eligible recipients, including those without a recorded delivery or reply. Define exclusions before assignment.
- Supporting metrics: Clicks and replies are descriptive, not alternative routes to declaring a winner.
- Analysis: One two-sided two-proportion z-test at a 5% significance level, with an unpooled normal-approximation 95% confidence interval for B minus A. Check suitability before launch; do not select the method after inspecting results.
- Business threshold: Replacing A requires an estimated improvement of at least 0.5 percentage points, alongside the statistical rule and resolved operational concerns. This threshold does not establish that the true improvement exceeds 0.5 points.
Recruitment and observation
- Design: Fixed horizon; no interim winner decisions. All times use UTC.
- Recruitment: Every day from 1 through 14 June 2026. Allocate 600 recipients daily on 1–10 June and 500 daily on 11–14 June, split equally between arms: 8,000 total.
- Coverage rule: Recruitment must span the full fourteen-day schedule. Do not borrow future quotas, backfill missed quotas on later days, or close early because a cumulative target was reached.
- Reason: This preserves planned calendar coverage rather than concentrating recruitment on early days. It costs flexibility and does not guarantee balanced audience composition within each day.
- Outcome deadline: Each recipient retains seven full days from assignment. Recruitment closes at 23:59 on 14 June; the latest outcome window closes at 23:59 on 21 June.
- Reconciliation: 22 June, 09:00–12:00, before opening comparative results.
- Final analysis: 22 June, 14:00. Unresolved ingestion issues may delay analysis, but never extend recruitment or outcome deadlines.
- Shortfall: Fewer than 8,000 assignments means no confirmatory winner claim. Report estimates and uncertainty as exploratory; do not extend recruitment.
Owners and monitoring
- Operations owner: Campaign manager checks assignment totals, tracking health, recipient issues and capacity daily at 10:00, with urgent incidents handled when detected.
- Analyst: Checks allocation and completeness on 4 and 11 June at 12:00, without examining comparative primary outcomes.
- Decision owner: Marketing lead records the final decision after the scheduled analysis.
- Access practice: Restrict comparative outcomes until final analysis where practical. Log accidental exposure without changing the schedule.
Pause and recovery rules
- Immediate recipient pause: A confirmed opt-out handling failure or audience permission problem pauses affected activity for investigation.
- Integrity pause: Confirmed cross-arm assignment of one recipient pauses recruitment while scope is established.
- Tracking pause: A failed end-to-end check triggers investigation; confirmed loss of outcome capture pauses recruitment.
- Accuracy pause: A confirmed incorrect offer or materially misleading statement pauses affected activity.
- Capacity pause: More than 100 unresolved campaign-related conversations at two consecutive hourly checks pauses new recruitment for capacity review. This is a hypothetical local threshold.
- Restart authority: The operations owner may authorise an operational restart after documenting the remedy. Restart does not authorise a winner claim.
- Fixed calendar: Recruitment and outcome deadlines never extend. Already assigned recipients retain their original outcome windows. Missed daily quotas remain unfilled.
- Recoverable data: Ingestion delays may be repaired using original event times. Include only events occurring within each original outcome window, not later events received before analysis.
- Material deviation: Changes to treatment, assignment, eligibility, scheduling rules or follow-up end confirmatory interpretation. A restart requiring those changes proceeds only as operational activity or a separately planned experiment.
- Reporting: Record incident times, affected recipients, missing data, remedies and every deviation. Label any analysis after material deviation or sample shortfall exploratory; report the original assigned-group estimate where recoverable and identify any subset analysis separately.
Final decision table
Apply these outcomes only if the planned sample, observation and integrity conditions hold. Statistical significance below means a two-sided p-value below 0.05.
- Adopt B: B minus A is positive and statistically significant, the estimated gain is at least 0.5 percentage points, and operational concerns are resolved.
- Retain A on performance evidence: B minus A is negative and statistically significant. Report that A performed better under the planned comparison, even if the estimated difference is smaller than 0.5 points.
- No performance-winner claim: The difference is not statistically significant, even if the estimated gain exceeds 0.5 points. Retain A as the operational default.
- No rollout-qualified winner claim: B is significantly better but its estimated gain is below 0.5 points. Report the statistical difference and unmet business threshold; retain A.
- No confirmatory claim: Sample shortfall, material deviation or unresolved data integrity prevents the planned interpretation. Retain A unless a separately documented operational reason requires another choice.
5. Work through the final comparison
Hypothetical completed example: Assume the plan ran without deviation, all outcomes matured, and each arm contained 4,000 assigned recipients. A recorded 160 purchasers and B recorded 200.
- A purchase rate: 160 / 4,000 = 4.0%.
- B purchase rate: 200 / 4,000 = 5.0%.
- Estimated difference: B minus A = 1.0 percentage point.
- Planned test: The two-sided pooled z-test gives approximately p = 0.031.
- Uncertainty: The unpooled normal-approximation 95% confidence interval is approximately +0.09 to +1.91 percentage points.
- Decision: Adopt B, provided operational concerns are resolved.
The estimate exceeds the business threshold, but the interval includes improvements below 0.5 points. The result therefore does not establish that the true improvement exceeds that threshold. Nor does it guarantee the same result in another audience or campaign.
A favourable click result could not rescue an inconclusive purchase result. Likewise, an operational pause would not make the unaffected variant the winner.
6. Handle pauses and previous peeking honestly
The filled plan permits recovery from delayed ingestion without changing exposure or observation periods. It does not permit rescheduling missed sends, rewriting the offer or extending follow-up while preserving the original confirmatory claim. Those changes alter the experiment.
If comparative results have already been checked repeatedly, preserve the original plan and reconstruct what happened: which outcomes and segments were inspected, when, and whether those observations changed activity.
Stop further result-driven revisions while continuing necessary recipient protections. Ask an appropriately qualified analyst whether the actual procedure still supports a valid analysis. Do not select a convenient past checkpoint as the supposedly planned endpoint.
Waiting longer does not automatically repair opportunistic stopping. Neither does applying a sequential label retrospectively. When confirmatory interpretation is unsupported, report exploratory findings and use them to plan a fresh test.
7. Close with a decision record, not just a winner label
Save the plan version, actual recruitment dates, daily allocations, observation cutoffs, achieved sample, incidents, deviations, effect estimate, interval and decision. Include who authorised restarts and whether anyone accessed comparative outcomes early.
Keep three statements separate: what the data suggest, what the planned method supports, and what the business chooses. Retaining A by default is not proof of equivalence. Choosing a simpler message for operational reasons is not an experimental win.
Before launch, have the operations owner, analyst and decision owner approve the completed record. The practical test is whether another colleague could apply its rules after a pause or ambiguous result without inventing a new stopping rule.
Put the next conversation to work.
Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.
Explore Communications