Back to all articlesCampaign measurement

Choose the right randomisation unit for a campaign experiment

Alactic Editorial10 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

If two colleagues receive different campaign variants but discuss the same purchase, they are not isolated experimental subjects. One may forward an offer, explain a message or persuade the other to respond. Your recipient-level comparison then measures a mixture of direct treatment and shared influence.

Assign the experiment at the smallest defensible group that contains the important influence you expect. Use recipient-level randomisation when recipients can reasonably experience and act on their assigned treatment independently. Use account-, household- or another group-level assignment when the treatment or decision is likely to spread within that group.

The trade-off is real: larger groups can reduce cross-arm contamination, but they leave fewer independent units. A campaign with many contacts may still have only a modest number of independently assigned accounts. Choose the unit before launch, and make the analysis respect that choice.

1. Locate the interference before choosing the unit

Start with a concrete question: How could someone assigned to one arm change the experience or outcome of someone assigned to the other? Do not choose account-level assignment merely because account IDs are convenient. Identify the mechanism you are trying to contain.

  • Shared information: recipients forward messages, compare prices or discuss a deadline.
  • Shared decisions: several contacts contribute to one purchase, booking or renewal.
  • Shared resources: a household uses one budget, or a team has one allocation of seats.
  • Shared service behaviour: a representative learns a treatment-specific approach and applies it to control recipients too.

These mechanisms suggest different boundaries. An account may contain a buying committee but not a representative's entire portfolio. A household may contain a shared spending decision but not a wider referral network. A branch may be more appropriate than a parent company if branches act independently; it may be too narrow if headquarters sets the decision.

Write the suspected path in one sentence: “Contacts within the same account discuss the offer before making a single purchasing decision.” That sentence is a stronger basis for the assignment unit than “We usually split contact lists.”

Some interference cannot be contained neatly. If recipients regularly share offers across businesses, account-level assignment reduces one problem without eliminating all spillover. Record that limitation rather than describing the accounts as fully isolated.

2. Separate assignment, exposure and outcome units

Three units can differ without making the design wrong:

  • Assignment unit: the entity randomly allocated to an arm, such as an account.
  • Exposure unit: the entity that receives or encounters the campaign, such as an individual contact.
  • Outcome unit: the entity on which success is measured, such as an account with a qualifying purchase.

For an account-randomised campaign, individual contacts can receive messages while the account remains the assignment unit. An account outcome should not be copied onto every contact row and then treated as several independent successes.

Also distinguish assignment from delivery. A contact assigned to a treatment may never receive the message. A control contact might encounter it through a colleague. Neither event changes the original random assignment.

Define the decision you want the experiment to support. “Does adopting this campaign increase the share of accounts that purchase?” points toward an account-level outcome. “Does this message change an individual's response?” may support a recipient-level outcome, but shared influence still determines whether individual assignment is defensible.

The outcome unit alone does not settle the assignment unit. A personal reply can be influenced by a colleague, and an account purchase can involve multiple separately exposed people.

3. Use a filled assignment decision table

Fictional worked example: All counts, allocations and thresholds in this example are hypothetical, not customer results. A team plans to compare its usual campaign, Arm A, with a revised explanation of the offer, Arm B. The eligible population contains 120 business accounts, each with five eligible contacts: 600 recipients in total. Contacts within an account discuss purchases, and the main outcome is whether the account makes a qualifying purchase during a predeclared observation period.

The following filled decision table compares two candidate designs. It is a planning artifact, not a power calculation.

Decision field          Recipient-level option       Account-level option
Assignment unit         Individual contact           Business account
Eligible population     600 contacts                 120 accounts; 600 contacts
Planned allocation      300 contacts A; 300 B         60 accounts A; 60 B
Contacts per arm        300 each                     300 each in this example
Within-account arms     May be mixed                 One arm per account
Main interference risk  Colleagues share variants    Sharing across accounts
Primary outcome         Account purchase is awkward  Account purchase aligns
Independent assignment  600 contact assignments      120 account assignments
Analysis limitation     Shared decisions undermine   Contacts are not 600
                        a simple contact comparison  independent observations
Decision                Reject for this question     Prefer, subject to power
                                                     and boundary checks

The contact total is identical in both options, but the experimental information is not. Recipient-level assignment creates more randomised assignments; it does not make shared purchasing decisions independent. Account-level assignment contains the expected discussion within each arm, at the cost of fewer independently assigned units.

Here is an illustrative allocation extract. Each account has five contacts; the recipient-level column shows how a mixed assignment could appear.

Account   Recipient-level allocation   Account-level allocation
AC-01     3 A / 2 B                    All 5 in A
AC-02     2 A / 3 B                    All 5 in B
AC-03     4 A / 1 B                    All 5 in B
AC-04     1 A / 4 B                    All 5 in A
AC-05     3 A / 2 B                    All 5 in A
AC-06     2 A / 3 B                    All 5 in B

This extract illustrates the designs; it is not a method for constructing random allocations manually. In the preferred design, the team would randomly allocate the full account roster, then apply each account's assignment to its eligible contacts.

4. Count independent groups, not just reachable contacts

Group-level assignment is not a free improvement. Responses within a group may be similar because members share circumstances, needs or decision-makers. Adding more contacts to an existing account may add less information than adding a new independent account.

For the hypothetical campaign, reaching all 600 contacts does not turn the account-randomised experiment into a study with 600 independent units. There are 120 randomised accounts. The precision of the comparison depends on the number of accounts, their outcome variability, their sizes and the similarity of responses within them.

Unequal group sizes make the choice harder. If a few large accounts contain most recipients, a balanced account count can still produce very different contact totals between arms. Large accounts may also dominate a recipient-weighted outcome.

Before allocation, inspect the distribution of group sizes and relevant baseline characteristics. If those characteristics are likely to matter, consider randomising within a small number of predeclared strata, such as account-size bands. Avoid creating so many strata that allocation becomes constrained or difficult to analyse.

Ask for a power assessment using the intended assignment unit, expected outcome frequency, meaningful effect size and plausible within-group similarity. Do not reuse a recipient-level sample-size estimate unchanged for an account-level design. If the group count is too small, the honest options include recruiting more groups, changing the question or accepting that the result may be inconclusive.

5. Lock the assignment rules before sending

The design needs a short assignment specification that campaign operators and analysts can both follow. This is a recommended operating document, not a claim about a particular product control.

  1. Define the eligible roster. State which recipients and groups enter the experiment and the date on which eligibility is fixed.
  2. Define the group boundary. Explain why it contains the expected influence. Record known exceptions, such as purchasing decisions shared across accounts.
  3. Specify allocation. Name the arms, allocation ratio, randomisation method and any predeclared balancing factors. Retain a reproducible assignment record.
  4. Set rules for later changes. Decide how newly eligible contacts, account changes and uncertain group membership will be handled without reference to emerging results.
  5. Specify treatment consistency. For group assignment, define how all eligible members remain in the group's arm and how other campaigns will avoid introducing the competing treatment.
  6. Fix the outcome and observation period. Define qualifying events, counting rules and the end of follow-up before examining arm performance.

A useful default for a fixed-cohort experiment is to keep newly eligible groups outside the primary cohort. New contacts in an already assigned group can inherit its arm under a predeclared rule, while their inclusion in exposure and outcome reporting is specified separately.

This article assumes defensible group membership is available; resolving identities and deciding contact ownership are separate tasks. If uncertain membership could change the appropriate assignment, resolve the design uncertainty or narrow eligibility before launch rather than improvising during the experiment.

6. Match the analysis to the assignment and the question

For the fictional account-randomised design, a straightforward primary analysis compares the share of assigned accounts with a qualifying purchase in Arm A against the share in Arm B. Each account contributes once. This answers an average-account question rather than giving larger accounts more weight because they have more contacts.

If the business instead wants an average-recipient outcome, define that target explicitly. Recipient outcomes within an assigned account remain clustered. Use an analysis that accounts for that clustering rather than a simple test that treats every recipient as independent. With few groups, inference needs particular care and may require specialist statistical support.

Keep the primary comparison based on original assignment. Removing accounts because nobody received a message, or comparing only contacts who opened or replied, conditions the analysis on events that occur after assignment. Those events can be affected by treatment and can make the retained groups less comparable.

Report delivery and exposure separately as process measures. They help explain what happened, but a delivered-only comparison is not automatically a causal estimate of the treatment's effect on delivered recipients.

In the results, show assigned group counts, group-size distribution, outcome counts, the estimated difference and uncertainty from an appropriate analysis. Record spillover evidence alongside them. An apparently small difference with wide uncertainty is not evidence that the variants are equivalent.

7. Recover from contamination without rewriting the experiment

Suppose the hypothetical campaign review finds that some contacts in Arm A received the Arm B message. Preserve the original allocation and document the mismatch. Do not quietly relabel those accounts as B or remove them because their exposure is inconvenient.

Use a simple incident classification:

  • Execution mismatch: the campaign sent a treatment inconsistent with the recorded assignment. Correct future execution, preserve the exposure history and assess how widespread the problem is.
  • Within-group spillover: colleagues discuss or forward the message inside an assigned account. This is part of the group-level treatment environment the design intended to contain.
  • Cross-group spillover: treatment information reaches another assigned group. Record the evidence and assess whether the assumed boundary remains credible.
  • Shared operator behaviour: staff apply treatment-specific language or handling across arms. Review whether the actual intervention now differs from the planned one.

When possible, collect spillover information consistently across arms rather than investigating only disappointing results. Report known contamination and uncertainty about unobserved sharing.

If the problem is limited, retain the original-assignment analysis and label any additional sensitivity analyses clearly. If the arms no longer represent distinguishable campaign policies, the experiment may not answer the intended question. Pause further exposure where appropriate, document the failure and redesign. A later move to account-level assignment cannot retroactively repair an earlier recipient-randomised experiment; treat it as a new experiment.

8. Make the assignment decision before the next launch

Prepare a one-page decision record using the filled example as a model. It should name the interference mechanism, proposed group boundary, eligible group count, arm allocation, primary outcome, analysis unit and unresolved limitations.

Then run an allocation rehearsal without sending messages. Measure the number of independent groups per arm, recipients per arm, group-size imbalance, unresolved memberships and groups that would contain conflicting assignments. Under an account-level design, conflicting assignments within an account should be corrected before launch.

Finally, ask one approval question: Would this design still answer our business question if colleagues shared everything they received within the proposed group? If yes, and the independent group count supports a useful comparison, the unit is defensible. If not, widen the boundary, narrow the question or postpone the experiment until a credible comparison is possible.

Campaign measurementExperiment designRandomisationAccount-level analysis

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications