Back to all articlesCampaign measurement

Decide whether a campaign test has enough evidence to act

Alactic Editorial9 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

In a hypothetical campaign test, a variant gets a 12% reply rate while the control gets 10%. In this fictional scenario, a reporting screen puts the larger number first, and someone calls it the winner. But with only 400 recipients per group, that two-percentage-point lead is too uncertain to establish a meaningful improvement.

A better decision starts before the send: define the smallest improvement worth acting on, assess whether the available audience can resolve it, and agree how uncertainty will affect the decision. Afterward, compare an interval estimate with that threshold—not just one observed rate with another.

The outcome might be a supported launch, a separately planned confirmatory test, or an inconclusive result. A test does not owe the team a winner.

1. Define what would make the change worth using

Your minimum worthwhile effect is the smallest improvement that justifies switching from the control to the variant. Express it in the same units as the outcome, preferably absolute percentage points for a reply-rate test.

Hypothetical example: a team decides that switching requires at least two additional qualifying replies per 100 assigned recipients. Its minimum worthwhile effect is +2 percentage points. Moving from 10% to 12% is a two-percentage-point increase, or a 20% relative increase. Both describe the same change; neither establishes how precisely it has been measured.

Choose the threshold from operating consequences: the effort to maintain the campaign, capacity to handle additional replies, and whether those replies advance the objective. More low-value conversations may increase workload without improving the outcome that matters.

  • Primary outcome: the measure used to decide the test.
  • Minimum worthwhile effect: the improvement needed to justify switching.
  • Guardrails: outcomes that must not deteriorate unacceptably, such as opt-outs or unresolved conversations.
  • Decision deadline: when the business needs an answer.

Do not lower the threshold after seeing an attractive result. If costs or priorities genuinely change, record the revised business decision separately rather than presenting it as success against the original test plan.

2. Make the comparison interpretable

An interval estimates sampling uncertainty under assumptions. It does not repair an unfair comparison. Randomly assign eligible recipients to control and variant, keep assignment stable, and avoid exposing the same person to both versions during the experiment.

Change one intended campaign element where possible. Keep timing, eligibility, follow-up handling, and measurement rules comparable. Otherwise, a difference attributed to copy might reflect a different send day or a faster response team.

Define the denominator before launch. In the fictional worked example below, reply rate means unique recipients with a qualifying reply divided by all eligible recipients randomly assigned to that group. A qualifying reply is a relevant response to the campaign question; opt-out requests are tracked separately. Each recipient gets the same hypothetical seven-day observation period.

Using assigned recipients keeps delivery failures visible in the campaign outcome. Replies per delivered message can be a secondary diagnostic, but conditioning the main result on delivery can weaken comparability if the variant affects delivery. Record assigned, attempted, delivered, and replied counts separately where reliable records are available.

For WhatsApp campaigns, eligibility must respect recipient permission and opt-outs. Approved templates are used to initiate Platform conversations and outside the 24-hour customer-service window, which opens or resets with a user message. Template approval does not guarantee delivery. Consult the current WhatsApp Business Messaging Policy for precise requirements, and distinguish campaign preparation, template review, and actual delivery.

3. Check whether your audience can answer the question

Small samples are not automatically useless. They can identify large differences while remaining poorly suited to resolving small ones. The planning question is not, “How many recipients make a test valid?” It is, “How narrow must the uncertainty be for this decision?”

Hypothetical planning illustration: with 400 recipients per group and reply rates around 10%–12%, a conventional approximate 95% interval for the difference has a margin of roughly four percentage points. That is wider than a minimum worthwhile effect of two percentage points. A result near the threshold will therefore be difficult to classify.

For equally sized, independent groups, a rough planning margin is 1.96 × sqrt(2 × p × (1 − p) / n), where p is an assumed reply rate and n is the number in each group. This is a precision approximation, not a universal sample-size rule.

Using a hypothetical assumed rate of 11%, about 7,500 recipients per group gives a margin near one percentage point. That targets interval width; it does not guarantee detecting an improvement or establishing that it exceeds the worthwhile threshold. A full planning calculation should consider the effect you expect and the probability of meeting your decision rule.

If only 800 eligible recipients are available in total, acknowledge the limitation before sending. You might test a more substantial change, plan comparable future cohorts in advance, or treat the campaign as exploratory learning. A different report label cannot make a narrow audience answer a fine-grained question.

4. Read the interval against the business threshold

The observed difference is a point estimate. An interval shows its precision under the chosen method. For the hypothetical examples here, the approximate interval is:

Difference ± 1.96 × sqrt[pA(1 − pA)/nA + pB(1 − pB)/nB]

The difference is variant rate minus control rate. Rates enter as proportions; results are converted to percentage points. This approximation illustrates the worksheet with the counts below. Sparse replies, clustered assignments, or dependent observations require a method appropriate to that design.

A frequentist 95% confidence interval does not mean this particular interval has a 95% probability of containing the true effect. The procedure is designed to cover the true effect in about 95% of repeated comparable experiments when its assumptions and sampling plan hold.

For a conservative launch rule, compare the entire interval with the minimum worthwhile effect:

  • Lower bound above the worthwhile threshold: evidence supports an improvement large enough to justify switching, subject to guardrails.
  • Interval crosses the worthwhile threshold: the business question remains unresolved, even if the interval excludes zero.
  • Upper bound below the worthwhile threshold: the evidence does not support a worthwhile gain; retaining the control is usually more defensible for this objective.

An interval entirely above zero can support some improvement without establishing enough improvement. An interval crossing zero does not prove equivalence. A narrow interval below the worthwhile threshold can also be useful evidence, rather than merely a failed attempt to find a winner.

5. Use this filled evidence worksheet

All figures and scenarios in this worksheet and the comparisons below are hypothetical, not Alactic customer results. The predeclared rule is to launch only when the interval's lower bound exceeds +2 percentage points and guardrails are acceptable. Reuse these fields for another test, replacing the assumptions before launch and filling observed results afterward.

Completed worksheet: promising but imprecise

  • Question: does a more specific opening question increase qualifying replies enough to justify switching?
  • Assignment: independent recipient-level random assignment; one version per recipient.
  • Primary outcome: unique qualifying replies within seven days divided by assigned eligible recipients.
  • Analysis endpoint: after all 800 assigned recipients complete their observation period; no early stopping for a favorable reply result.
  • Control: 400 assigned recipients; 40 qualifying replies; observed rate 10.0%.
  • Variant: 400 assigned recipients; 48 qualifying replies; observed rate 12.0%.
  • Observed difference: +2.0 percentage points.
  • Minimum worthwhile effect: +2.0 percentage points.
  • Hypothetical guardrail review: no predefined guardrail threshold was crossed; rare harms remain uncertain at this sample size.

The calculation uses sqrt[(0.10 × 0.90 / 400) + (0.12 × 0.88 / 400)], giving a standard error of about 0.0221, or 2.21 percentage points. Multiplying by 1.96 gives a margin of about 4.33 percentage points. The approximate 95% interval is therefore −2.3 to +6.3 percentage points.

  • Interpretation: the interval includes a decline, no improvement, and a worthwhile improvement. Reaching the target with the point estimate does not establish it.
  • Decision: retain the control; do not declare a winner.
  • Next step if sufficient audience is feasible: plan a separate confirmatory test and analyze it on its own.
  • Next step if the audience is exhausted: close as inconclusive and preserve the record.

Launch comparison: a different hypothetical result

A separately planned test has 2,000 recipients per group. The control receives 200 qualifying replies, or 10%, and the variant receives 300, or 15%. The difference is +5.0 percentage points, with an approximate 95% interval of +3.0 to +7.0 percentage points.

With the same +2-point threshold, this supports launch if guardrails and data checks pass. The decisive feature is not just the higher observed rate: the interval's lower bound clears the worthwhile threshold.

Positive does not necessarily mean worthwhile

In another hypothetical test with 2,000 recipients per group, 200 control replies and 240 variant replies give rates of 10% and 12%. The approximate interval for the +2-point difference is +0.1 to +3.9 percentage points.

This supports a positive effect under the stated method, but does not establish that it exceeds +2 points. A confirmatory test is worth considering only if resolving that uncertainty justifies the additional audience and time.

6. Repeat without creating a winner hunt

The straightforward repeat option is a separately planned confirmatory test analyzed on its own. Use the first result for planning, define the new audience and sample size before starting, and keep the new result distinct from the exploratory finding. Retain the control meanwhile unless a separately documented business decision justifies a change.

There is an important difference between planning future cohorts before examining any results and deciding to extend an experiment because its first result looks promising. Extending or pooling a test after inspecting its results requires an analysis that accounts for the data-dependent continuation. Writing down a new endpoint before the next batch does not, by itself, restore the nominal coverage of an ordinary fixed-sample confidence interval.

If combining old and new data is necessary, obtain an analysis plan appropriate to that selection and continuation process rather than applying the worksheet formula uncritically. Do not keep commissioning separate tests until one produces an attractive interval and then report only that success; selection across tests also changes the evidential picture.

Likewise, avoid checking a fixed-sample interval after every few replies and stopping at the first favorable result. Use the planned endpoint or a design that explicitly supports interim decisions. If the message, eligibility rules, or reply definition changes, record that you are testing a new treatment or question.

When the audience cannot grow enough, narrow the claim rather than inflate it. Qualitative review may suggest a stronger future variant, but it cannot retroactively make this numerical comparison decisive.

7. Record one accountable decision

Separate the estimate, uncertainty, and action. “The variant was two points higher” is an observation. “The interval includes a decline and a worthwhile gain” describes uncertainty. “Retain the control because the launch threshold was not met” is a decision.

A business may choose a reversible rollout with weaker evidence when switching costs are low. That is a risk decision, not stronger statistical proof. Record the trade-off, monitoring owner, and rollback conditions instead of relabeling the test a confirmed win.

Before your next campaign, fill in the eligible audience, primary outcome, observation window, worthwhile effect, and analysis endpoint. Estimate the likely interval width. If it is much larger than the improvement you care about, change the evidence plan before using the audience—not after the report disappoints.

Campaign measurementA/B testingDecision-making

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications