Compare campaigns fairly when recipient mixes differ
Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.
Campaign A produced a higher reply rate than Campaign B. Before calling its approach more effective, check who received each campaign. If A reached mostly people who had already engaged with your business, while B reached mostly people who had not, the overall rates answer different questions.
To compare campaign performance across different audiences, first inspect audience composition, then compare results within relevant pre-campaign groups. If both campaigns have usable observations in those groups, calculate a common-weight comparison: apply the same audience mix to each campaign’s group-level rates. This creates a more comparable descriptive summary. It does not establish that the campaign approach caused the difference.
The worked example below shows how an apparent winner can change after accounting for prior engagement—and how to decide whether that finding is enough for reporting or needs a randomised test.
1. Separate actual results from comparative effectiveness
Two questions often get collapsed into one dashboard number:
- What happened in the audience we actually contacted? The observed overall rate answers this.
- How do the campaigns compare after holding a measured audience characteristic constant? Group-level rates and a common-weight summary help answer this.
Neither question is a substitute for the other. Actual reply counts matter for workload and campaign accounting. A standardised rate is useful for understanding composition, but it is not an additional set of replies and should not replace the observed result.
A third question requires stronger evidence: Would changing the campaign approach improve outcomes for comparable recipients? Historical campaign cohorts may differ in timing, offer, relationship history, or other characteristics. Adjusting for prior engagement alone does not remove those differences.
Before calculating anything, write down which decision the comparison supports. Explaining last month’s totals may need only descriptive analysis. Choosing a new default message approach may justify a prospective randomised test.
2. Lock the metric and the pre-campaign grouping
A fairer comparison starts with consistent measurement. Use the same outcome definition, denominator, observation window, and counting rules for both campaigns.
For the fictional example in this article, the measurement specification is:
- Unit: one unique recipient within each campaign cohort.
- Denominator: recipients recorded as delivered the campaign message.
- Outcome: at least one recipient reply within seven days of that recipient’s delivery.
- Counting: each recipient contributes at most one reply outcome per campaign, regardless of message count.
- Previously engaged: at least one recorded inbound reply during the 90 days before the campaign’s defined start.
- No recorded prior engagement: no such reply in that lookback period, with adequate historical coverage.
These are hypothetical analytical choices, not universal measurement rules. A reply is also not necessarily a positive commercial outcome: an opt-out or a complaint can be a reply. If the business question concerns qualified interest, define that outcome separately and apply the same classification rules to both cohorts.
Freeze group membership using information available before the campaign. Do not classify someone as previously engaged because they replied to the campaign being evaluated. That would use the outcome to define the comparison group.
Check whether historical coverage is genuinely comparable. Missing history is not evidence of no engagement. If one cohort has incomplete records, retain an unknown-history group or restrict the analysis to comparable coverage, then disclose the restriction.
3. Filled fictional example: the overall winner changes within groups
All campaigns, counts, rates, and time windows in this worked example are hypothetical. They are not Alactic customer results. Assume the observation windows are complete, records are consistent, and both campaigns have usable data in each engagement group.
Observed campaign results — fictional worked example Campaign A Previously engaged: 800 recipients; 160 replied; 20.0% No recorded prior engagement:200 recipients; 10 replied; 5.0% Total: 1,000 recipients; 170 replied; 17.0% Previously engaged share: 80.0% Campaign B Previously engaged: 200 recipients; 50 replied; 25.0% No recorded prior engagement:800 recipients; 64 replied; 8.0% Total: 1,000 recipients; 114 replied; 11.4% Previously engaged share: 20.0% Observed overall gap, A minus B: +5.6 percentage points Within previously engaged, B minus A: +5.0 percentage points Within no recorded prior engagement, B minus A: +3.0 percentage points
On the aggregate result, A appears stronger: 17.0% versus 11.4%. Yet B has the higher observed reply rate in both prior-engagement groups.
The composition explains this apparent contradiction. A’s audience is mostly previously engaged recipients, whose observed reply rates are higher in both campaigns. B’s audience is mostly recipients without recorded prior engagement. The overall rate combines performance within groups with how heavily each group is represented.
The defensible findings are therefore separate: A generated more replies in its actual cohort; B recorded higher rates within both measured engagement groups. Neither statement establishes why those differences occurred.
4. Apply common weights to create a comparable summary
A common-weight comparison applies one audience composition to both campaigns’ group-level rates. For each campaign, multiply each group’s rate by its chosen weight, then add the results.
Here, pooling the two equal-sized cohorts produces a hypothetical reference audience that is 50% previously engaged and 50% without recorded prior engagement. Use those same weights for both campaigns.
Common-weight comparison — fictional worked example Reference mix Previously engaged: 50% No recorded prior engagement: 50% Campaign A (0.50 × 20.0%) + (0.50 × 5.0%) = 12.5% Campaign B (0.50 × 25.0%) + (0.50 × 8.0%) = 16.5% Standardised gap, B minus A: +4.0 percentage points Observed rates: A 17.0%; B 11.4% Common-weight rates: A 12.5%; B 16.5%
The standardised result describes what the weighted average would be if each campaign’s observed group rates were combined using that reference mix. It is a calculation, not a prediction that either campaign would reproduce those rates in a new audience.
Choose weights for a reason. A pooled audience is useful when summarising the combined historical cohorts. A documented planning audience may be more relevant to an upcoming decision. Equal weights can make group comparisons easy to inspect, but they may not represent the business’s actual recipient mix.
Record the reference population and weights before interpreting the result. Do not try several weighting schemes and publish only the one that favours a preferred campaign. If multiple mixes are relevant, show them as a sensitivity analysis, clearly labelled.
In this example, B’s observed rate is higher in both groups, so any common mix that gives positive weight to either group also favours B descriptively. If one campaign performed better in one group and worse in another, the choice of weights could change the summary winner. In that case, the group-level results may be more useful than a single score.
5. Check whether the comparison is supported by the data
Standardisation is straightforward arithmetic, but a reliable interpretation depends on the underlying records. Before publishing, work through these failure cases.
A group is missing or too small
If a campaign has no previously engaged recipients, its rate for that group cannot be calculated from that campaign. Do not substitute zero: no observations is different from an observed zero-response rate. Report the unsupported comparison and gather comparable data before producing a full common-weight result.
Small groups can also produce unstable rates. Show numerator and denominator beside every percentage. Where the decision is consequential, obtain an appropriate uncertainty assessment rather than treating a small numerical difference as decisive.
The outcome windows are unequal
A campaign observed for a full week cannot be fairly compared with one sent recently if late replies still count toward the chosen outcome. Wait for both windows to close, or use the same shorter window for both and label it explicitly.
The denominator hides another difference
This example measures replies among delivered recipients. That is useful for one question, but delivery is a post-send event. If delivery differs across campaigns, the delivered audiences may themselves be selected differently. Keep delivery counts and the broader campaign funnel alongside the reply analysis; do not use a delivered-recipient rate alone to claim end-to-end superiority.
Recipients or exposures overlap
A person exposed to both campaigns may carry an effect from the first into the second. Repeated contacts also make observations less independent. Inspect overlap, ordering, and other contacts during the outcome window. If contamination is substantial, narrow the reporting claim rather than presenting the cohorts as clean alternatives.
The groups conceal important differences
Two previously engaged audiences may still differ in how recently or frequently they replied. Add another pre-campaign dimension only when it is relevant, measured consistently, and supported by enough observations. Excessively detailed grouping can leave sparse cells and a misleading impression of precision.
6. Use standardisation for description, not causal attribution
The strongest appropriate summary of the fictional analysis is:
Campaign A had the higher observed overall reply rate. Campaign B had higher reply rates in both recorded prior-engagement groups and a higher common-weight rate using the pooled audience mix. These descriptive results account for the recorded engagement grouping, but do not isolate the effect of campaign approach.
That qualification matters because prior engagement is only one measured characteristic. Campaigns may have run at different times, used different offers, or reached recipients with different needs within the same group. Staff response practices and concurrent activity may also differ.
A common-weight comparison is useful when the immediate problem is an aggregate report that masks audience composition. It can identify whether an apparent performance lead persists within relevant groups and make reporting less misleading.
It cannot recover information that was never recorded or turn separately selected cohorts into a controlled experiment. Even a precise standardised difference can remain confounded. Avoid statements such as “B increased replies by four percentage points.” In this example, the supported statement is that B’s standardised observed rate was four percentage points higher.
7. Use a randomised test when the decision needs an effect estimate
If the business needs to choose between approaches rather than describe existing cohorts, design a prospective comparison in which comparable recipients are randomly assigned to alternatives before exposure.
A practical design can randomise within the same pre-campaign engagement groups. That helps balance the group mix while preserving a view of whether results differ between groups. Random assignment does not replace recipient permission or permission to contact; those remain separate operating requirements.
- Specify the change. If testing message wording, keep the offer and other operational conditions aligned where practical. If testing an entire campaign package, state that the result concerns the package, not one component.
- Define the primary outcome and denominator in advance. Choose an outcome tied to the decision, not simply the easiest metric to increase. For the primary comparison, normally retain recipients in their assigned groups so post-assignment events do not silently redefine the audiences.
- Control exposure overlap. Keep assignment consistent for each recipient and document other campaign contacts that could affect the outcome.
- Plan the analysis before launch. Set the observation window, sample-size rationale, stopping rule, and treatment of missing outcomes. Seek statistical support where the stakes warrant it.
- Report uncertainty and execution differences. A randomised design supports stronger causal interpretation when implemented well, but incomplete records, crossover, or unequal handling can still weaken the conclusion.
If randomisation is not practical, retain the descriptive comparison and explain its limits. Do not relabel historical adjustment as a test.
8. Make the next report auditable
For the next pair of campaigns, produce one comparison sheet with the measurement definition, audience shares, group counts, group rates, observed totals, reference weights, and standardised results. Keep unknown-history records visible and record any exclusions.
Use the fictional artifact above as the layout, replacing its values only after checking that both cohorts have complete outcome windows and consistent pre-campaign history. Have a second reviewer reproduce the weighted calculation from the group counts.
The measurable next action is simple: before the next performance review, complete that sheet for one campaign pair and label the decision it can support. If the decision is descriptive, report the observed and standardised results together. If the decision requires knowing which approach causes better outcomes, turn the unresolved question into a randomised test plan.
Put the next conversation to work.
Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.
Explore Communications