Back to all articlesCampaign measurement

Test one campaign change without confounding the result

Alactic Editorial10 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

A campaign variant introduces a stronger offer, shorter wording, a warmer audience, and a later send time. It generates more bookings. The team now has a result, but not an explanation: which change helped, and would it help again under the original conditions?

To test one campaign change without confounding the result, define the exact contrast before preparing the variants, randomly assign comparable recipients, and keep other relevant conditions consistent. If several changes must travel together, test them as a bundled treatment—but make the decision about that bundle, not its individual parts.

The practical starting point is an experiment brief that states what differs, what stays comparable, how success is measured, and which deviations would undermine the intended comparison. The filled fictional brief below shows how to make those commitments concrete.

1. Decide what the result must let you do

Start with the decision, not the two messages. A useful test question names a change and the action that evidence could support.

Hypothetical isolated-change question: Should the opening of our consultation invitation lead with the customer benefit rather than the service process?

Hypothetical bundled-treatment question: Should we replace our current campaign approach with a package that combines a new offer, message, and follow-up sequence?

Both questions are legitimate. They answer different business problems. The first can support a decision about the opening approach. The second can support a decision about the complete package, provided recipients are assigned fairly and execution is comparable.

The trouble begins when the team runs the second test but reports the first conclusion. A winning package does not establish that its wording caused the improvement. The offer might account for most of the difference, or the components might work only in combination.

Write the intended conclusion in advance: If the evidence supports B, we will adopt the benefit-led opening for this audience and offer. If that sentence claims more than the design can establish, narrow the claim or change the design.

2. Define one treatment, not necessarily one edited word

Testing one change does not mean changing only one word. A coherent treatment may require several related edits. Reframing an opening from process-led to benefit-led could change the first sentence, sentence order, and emphasis. Those edits can represent one conceptual contrast if that contrast is what the business needs to evaluate.

The boundary must be explicit. If the benefit-led message also adds a discount, removes an eligibility condition, and changes the call to action, it no longer isolates the opening approach.

Use a difference inventory before launch:

  • Intended difference: The message opening leads with the recipient benefit instead of the service process.
  • Necessary supporting edits: Limited phrasing changes needed to make each opening readable.
  • Unwanted differences: New incentive, different promise, changed eligibility, different destination, or extra follow-up.

Have someone who did not prepare the variants review this inventory against the actual materials. Ask them to list every difference they can see. An unexpected difference is easier to remove before assignment than to explain after results arrive.

This approach gives up a word-by-word causal claim in exchange for a more useful comparison of message approaches. That is a reasonable trade-off when the business would adopt the approach as a whole.

3. Filled fictional experiment brief: two message approaches

Fictional worked example: The business, audience, quantities, thresholds, and time periods below are hypothetical. They illustrate a recommended design, not customer results or software capabilities.

Decision and treatment

  • Business situation: A service business invites permissioned prospects to book an introductory consultation. The existing offer and booking process will remain unchanged.
  • Decision: Whether to use a benefit-led opening instead of a process-led opening in the next comparable campaign.
  • Testable question: Does assignment to the benefit-led invitation increase completed bookings compared with assignment to the process-led invitation?
  • Approach A: Opens by describing what happens during the consultation, then explains its benefit.
  • Approach B: Opens with the benefit of the consultation, then describes what happens.
  • Intended difference: The framing and order of the opening. This is a message-approach test, not a test of an individual word.

Audience and assignment

  • Eligible population: Prospects with appropriate permission, no opt-out, no existing consultation booking, and no other planned campaign contact during the observation period.
  • Unit of assignment: One prospect, with duplicate records resolved before allocation. Each prospect stays in the same arm.
  • Allocation: Randomly assign eligible prospects equally within recent-enquiry and older-enquiry groups, so each arm draws from both groups.
  • Planning quantity: A hypothetical capacity of 1,200 prospects, approximately 600 per arm. This is an operational example, not a claim that the test has enough statistical power.

Controlled conditions

  • Offer: Same consultation, eligibility, availability, and terms.
  • Message: Same sender identity, language, substantive information, call to action, and destination; similar overall length.
  • Timing: Both arms distributed across the same local-time sending blocks, rather than sending all of A before B.
  • Follow-up: Same planned contact rules, suppression rules, and stop conditions.
  • Service handling: Same booking capacity, routing rules, response guidance, and staffing coverage.
  • Measurement: Same event definition, observation window, deduplication rules, and reporting cutoff.

Measures and decision rule

  • Primary measure: Unique prospects completing a consultation booking within seven days of their scheduled initial send, divided by all prospects assigned to that arm.
  • Booking definition: A confirmed appointment record linked to the assigned prospect; multiple appointments count once. Cancellations are reported separately.
  • Diagnostics: Attempted sends, delivery status where available, replies, and booking-path failures.
  • Guardrails: Opt-outs, negative replies, and staff workload, assessed using the same definitions in both arms.
  • Practical threshold: A hypothetical two-percentage-point absolute booking-rate improvement is worth considering operationally.
  • Decision rule: Assess the estimated difference and its uncertainty against that threshold after the full observation period. Adopt B only if the evidence supports a worthwhile improvement without unacceptable guardrail deterioration; otherwise retain A or plan further evidence collection.

Deviations that undermine the intended comparison

  • One arm receives an additional incentive or a different booking destination.
  • Assignment is overwritten based on staff preference or perceived prospect quality.
  • A material execution error puts one arm outside the planned sending blocks.
  • A tracking fault makes bookings systematically harder to observe in one arm.
  • Staff use different follow-up or booking-priority rules by arm.
  • Recipients receive both approaches because assignment or duplicate handling fails.

The hypothetical allocation is not a sample-size recommendation. Before launch, use a relevant baseline booking rate, the smallest worthwhile effect, and the chosen uncertainty standard to assess whether the available audience can answer the question. If it cannot, label the exercise exploratory rather than promising a decisive winner.

4. Keep conditions comparable without demanding identical recipients

Recipients will differ in interest, availability, and likelihood of booking. Random assignment does not make individuals identical; it reduces systematic selection of one kind of recipient into one arm. Random variation can still produce imbalance, especially in a small test.

When an established characteristic strongly affects response, randomizing within a few predeclared groups can help maintain balance. In the fictional brief, enquiry recency is such a characteristic. Avoid creating so many tiny groups that assignment becomes fragile or difficult to audit.

Choose the assignment unit to match the risk of spillover. If several contacts from one business are likely to share the invitation, assigning each contact separately may expose the same buying group to both treatments. Assignment at the business-account level may be more appropriate, though it leaves fewer independent units for analysis.

Timing deserves similar care. Sending A in the morning and B in the evening mixes message approach with send time. Alternating or randomizing assignment within shared sending blocks helps prevent that avoidable difference. Keep a record of actual execution, not just the intended schedule.

For WhatsApp campaigns, permission and opt-outs remain eligibility constraints, not experimental variables. Approved templates are used to initiate Platform conversations and outside the customer-service window, which opens or resets with a user message. Check the current WhatsApp Business Messaging Policy for precise requirements. Preparation, template review, and actual delivery are separate states; template approval does not guarantee delivery.

5. Choose a denominator that preserves the comparison

A reply rate among delivered messages can be useful diagnostically, but it should not silently replace the primary measure. Delivery is observed after assignment and may differ between approaches. Restricting the analysis to delivered recipients can change which people are being compared.

The fictional brief therefore uses bookings per assigned prospect. This measures the effect of assignment to each approach under the tested operating conditions, including its delivery pathway. Report delivery information alongside it to help explain what happened.

Do not automatically treat a delivery difference as a confound. If both approaches were executed as planned and one experienced different delivery, that may be part of the approach's operational effect. If one arm was accidentally delayed or sent through a different setup, the intended contrast has changed.

Allow every prospect the full observation window before comparing outcomes. Looking at A after a complete week and B after only a day creates unequal opportunity to book. Also avoid declaring a winner whenever a live dashboard briefly looks favourable. Repeated informal checking followed by selective stopping can make a noisy difference look more convincing than it is.

6. Decide when a bundle is the better experiment

An isolated contrast is useful when the next decision concerns a component. A bundled treatment is useful when the business genuinely intends to deploy the package together.

Hypothetical example: A team is considering a new invitation that combines a different offer, a benefit-led message, and a revised follow-up process. If it would never deploy those elements separately, a randomized package comparison may answer the immediate decision more directly than several component tests.

The conclusion must remain at package level. Record every component and keep background conditions comparable. Do not report that the new wording won when the offer and follow-up also changed.

If learning about components matters, a later sequence of isolated tests may be easier to operate and interpret. Testing several factors simultaneously can investigate combinations, but it requires more design work and enough observations to assess the relevant effects. A small audience does not become more informative simply because it is divided into more variants.

7. Recover from deviations without rewriting the experiment

When execution breaks, preserve the original assignment record and document what happened, when, and to whom. Do not quietly remove inconvenient recipients or reclassify them into the message they eventually received.

  1. Classify the deviation: Did it alter treatment, exposure, outcome measurement, or only a minor administrative detail?
  2. Assess its scope: Identify affected recipients and whether the problem differed by arm.
  3. Separate the conclusions: The intended message comparison may be compromised even if the observed campaign outcomes remain useful operational evidence.
  4. Choose a recovery: Repair and restart when feasible, or report the compromised comparison with clear limitations. Any subset analysis should be labelled secondary and justified, not presented as the original test.

Never invalidate a test simply because its result is disappointing. A small difference or wide uncertainty is an outcome, not an execution failure. Conversely, a large apparent improvement does not repair a broken assignment process.

8. Make the next action auditable

Before the next campaign launch, complete one brief and ask a reviewer to check five things: the decision, intended contrast, assignment method, primary measure, and invalidating deviations. Resolve disagreements before anyone sees results.

Keep the approved materials, assignment record, execution log, and outcome definitions together. At readout, report the number assigned to each arm, primary outcome counts and rates, the estimated difference with uncertainty, guardrail outcomes, and any departures from the brief.

The immediate deliverable is not a winning variant. It is a comparison whose scope the team can defend: one named change, or one explicitly defined bundle, tested under conditions that make the resulting decision interpretable.

Campaign measurementA/B testingExperiment designMessaging operations

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications