Verify that an apparent campaign winner survives a confirmation test
Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.
The campaign report has a clear leader: one variant beat the baseline and every other message. Before the team replaces the baseline, ask a different question: would that same variant win on recipients who had no part in selecting it?
To confirm a campaign winner after testing multiple variants, separate discovery from proof. Freeze the selected variant, compare it with a defined baseline on a fresh eligible cohort, and decide the outcome using rules written before results arrive. Do not combine the exploratory and confirmation results to rescue a disappointing retest.
The worksheet below shows how to make that separation operational. All campaign details, counts, thresholds and results in this article are hypothetical, not customer results.
1. Treat the screening winner as a candidate
Screening asks which option deserves another test. Confirmation asks whether a specific, already-selected option improves a specific outcome against a specific baseline.
Those are different jobs. When a team screens many variants, it creates more opportunities for a result to look unusually good by chance. Even if the variants have identical underlying performance, their observed results will differ. Selecting the highest observed result selects both whatever real advantage it has and whatever favourable noise helped it reach the top.
This is multiple-comparison risk. It becomes harder to assess when the team also tries several audience cuts, conversion windows or success metrics and reports whichever combination looks best. The effective search is larger than the number of message variants alone.
Consequently, the selected uplift is often an optimistic estimate. A smaller uplift in confirmation is not automatically evidence that something broke. The original estimate may simply have benefited from selection.
A fresh confirmation test avoids asking the same observations to both nominate and validate the winner. It does not make the result certain or remove the need for sound measurement.
2. Complete the screening record before starting again
Fictional worked example: A team tested five campaign variants against its current baseline. Each arm contained 800 randomly assigned eligible recipients. The primary outcome was at least one completed booking within seven days of assignment, counted once per recipient.
The following filled worksheet preserves what happened without turning the best observed rate into a promise.
Worksheet A: exploratory screening record
- Business question: Which message should advance to a separate test for increasing completed bookings?
- Baseline: Current message and booking destination; 80 bookings from 800 assigned recipients, or 10.0%.
- Variant A: Shorter opening; 85 of 800, or 10.6%.
- Variant B: Different benefit emphasis; 88 of 800, or 11.0%.
- Variant C: Earlier placement of the booking link; 92 of 800, or 11.5%.
- Variant D: Benefit-led opening with an explicit booking instruction; 112 of 800, or 14.0%.
- Variant E: Question-led opening; 78 of 800, or 9.8%.
- Selected candidate: Variant D.
- Selection rationale: Highest observed primary-outcome rate, with no known tracking defect or operational failure that disqualified it.
- Exploratory estimate: D exceeded the baseline by 4.0 percentage points, equivalent to a 40% relative increase. Neither figure is treated as a confirmed improvement.
- Selection scope: Five variants compared with one baseline. Any additional segment, metric or time-window searches must be appended to this record.
- Status: Candidate selected; rollout not authorised by these results alone.
The rationale is deliberately modest. It does not claim that the opening caused the result, that every component of D helped, or that D will reproduce a 14.0% rate.
Archive the complete screening report, including losing variants. Otherwise, the organisation may later remember only the winning comparison and forget how many opportunities there were to find it.
3. Define what the confirmation test must establish
Write the decision rule before assigning the fresh cohort. Start with one primary outcome and a minimum improvement worth acting on.
For this fictional team, the primary outcome remains a completed booking within seven days of assignment. The smallest observed improvement that would justify rollout is 1.5 percentage points. That is a business threshold, not a universal statistical standard: the team must connect it to its own implementation costs, capacity and value per booking.
The test must also distinguish an improvement from sampling uncertainty. Here, the team requires a two-sided 95% confidence interval for the difference in booking rates whose lower bound is above zero. A positive point estimate alone is insufficient.
These are two separate gates:
- Evidence gate: Does the interval exclude no improvement?
- Practical gate: Is the observed improvement at least 1.5 percentage points?
This rule does not establish that the true improvement is at least 1.5 percentage points. To make that stronger claim, the team would need a rule requiring the interval’s lower bound to exceed that threshold, with a corresponding sample-size plan.
Secondary measures can explain the result, but they cannot replace the primary outcome after the fact. More clicks with no confirmed booking improvement is not a booking win.
4. Fill the fresh-cohort protocol
The following is a recommended operating design, not a statement about controls available in any particular product.
Worksheet B: confirmation protocol
- Candidate: Exact archived version of Variant D. No wording, destination or offer changes during the test.
- Control: Exact archived baseline version used in screening.
- Eligible population: Recipients meeting the same documented audience criteria, with the required channel permission, excluding opt-outs and anyone assigned to screening.
- Freshness check: Match against the screening assignment file using a stable recipient identifier; deduplicate before randomisation.
- Assignment unit: Recipient, with one assignment per person. If related recipients can influence one another, review whether a household or account-level design is needed instead.
- Allocation: Random 1:1 assignment, with both arms running across the same operating days and send periods.
- Planned size: 4,000 assigned recipients per arm, subject to analyst approval of the calculation before launch.
- Planning assumptions: Approximately 10% baseline conversion, a 2-percentage-point planning effect, a two-sided 5% significance level and approximately 80% power for detecting a positive difference. This does not mean 80% probability of passing both decision gates.
- Recruitment window: Up to 14 days. If recruitment finishes earlier, stop enrolling but retain every recipient’s full outcome window.
- Outcome maturity: Seven days after each recipient’s assignment. Final analysis waits until the last recipient’s window closes.
- Primary analysis: Difference in booking proportions using all assigned eligible recipients in their original arms, with a preselected confidence-interval method.
- Decision owner: Campaign lead, with measurement review before rollout authorisation.
- Change control: Any necessary material change pauses the protocol for review rather than silently creating a new variant.
The sample-size assumptions are not reusable defaults. Lower baseline rates, smaller effects, clustering or a different decision rule can require substantially different numbers. If the available audience cannot support the planned test, document that constraint before launch.
For a WhatsApp campaign, eligibility also requires attention to recipient permission and opt-outs. Approved templates are used to initiate Platform conversations and outside the customer-service window, which opens or resets with a user message. Template approval does not guarantee delivery. Check the current WhatsApp Business Messaging Policy when preparing the campaign, and keep preparation, template review and actual delivery as separate states.
5. Protect the comparison from operational changes
A fresh audience is necessary but insufficient. The candidate and baseline also need comparable treatment during confirmation.
Do not send the baseline early in the week and the candidate later, or give one arm faster follow-up. Random assignment cannot repair a systematic difference introduced after assignment. Keep offer terms, booking availability, destination behaviour and follow-up rules consistent unless they are explicitly part of the tested treatment.
Use assigned recipients as the primary denominator in this design. Restricting analysis to delivered messages, openers or clickers can select different groups after assignment and obscure the effect of deploying the campaign as a whole. Delivery and engagement rates remain useful diagnostic measures.
Before launch, test assignment logging and booking attribution. Record which failures invalidate the comparison. A broken booking event in one arm is different from a genuine increase in recipients who choose not to book.
During the test, inspect operational health without repeatedly declaring winners. If the team wants to stop early based on performance, it needs an appropriate sequential design established in advance. Otherwise, retain the fixed-sample analysis and do not stop simply when the result first looks favourable.
6. Apply the criteria even when the favourite disappoints
Worksheet C: hypothetical confirmation readout
- Baseline: 400 bookings from 4,000 assigned recipients, or 10.0%.
- Variant D: 440 bookings from 4,000 assigned recipients, or 11.0%.
- Observed difference: +1.0 percentage point.
- Approximate two-sided 95% interval: −0.34 to +2.34 percentage points, using an illustrative normal approximation for independent proportions.
- Evidence gate: Not passed; the interval includes zero.
- Practical gate: Not passed; the observed difference is below +1.5 percentage points.
- Decision: Do not replace the baseline on this evidence. Classify D as unconfirmed, not as proven ineffective.
The result remains compatible with both no improvement and a potentially useful improvement. It does not prove equivalence. The correct operating decision is narrower: the candidate did not satisfy the agreed rollout requirements.
For contrast, a hypothetical result of 500 bookings out of 4,000 for D against the same baseline would produce a +2.5-percentage-point difference, with an approximate interval of +1.12 to +3.88 percentage points. That would pass both stated gates, provided the data-quality checks also passed. It would still support an inference about the tested population and conditions, not a guarantee for future campaigns.
Do not pool the screening results with confirmation just to make D pass. Its screening performance helped select it; that selection cannot be ignored when interpreting a combined estimate.
7. Recover without turning another search into confirmation
When confirmation fails, choose the next action according to the failure, not the team’s attachment to the candidate.
- Valid but inconclusive test: Keep the baseline. Consider a separately planned follow-up only if the remaining uncertainty matters enough to justify the audience and operational cost. Do not casually add recipients until significance appears.
- Incomplete recruitment: Report the shortfall and resulting uncertainty. Do not lower the improvement threshold after seeing the outcome.
- Measurement or assignment failure: Document the defect, repair it and plan a new valid test. Do not label a compromised comparison a clean loss or win.
- Changed offer or campaign conditions: Identify what changed. A modified candidate answers a new question and needs a new protocol.
- Promising post-hoc subgroup: Treat it as a new exploratory hypothesis. Selecting a segment after the overall result recreates the multiple-comparison problem.
Fresh-cohort confirmation consumes audience and delays rollout. That trade-off is most worthwhile when a rollout is costly, difficult to reverse or likely to become the new organisational default. A small reversible campaign may justify a lighter process, but the evidence should still be labelled honestly.
8. Make the next decision auditable
Before the next campaign launches, complete the three worksheet records: screening, confirmation protocol and decision readout. Give each an owner, link the frozen campaign versions, and record the approval date before fresh-cohort assignment begins.
The immediate measurable action is simple: verify that every planned confirmation has a named baseline, an exclusion check for screening participants, one primary outcome, a sample-size rationale and written acceptance criteria. Any missing field is a reason to finish the plan before sending—not a reason to improvise after the results arrive.
Put the next conversation to work.
Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.
Explore Communications