Back to all articlesAudience segmentation

Investigate audience drift when the rules have not changed

Alactic Editorial10 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

Your recurring audience has changed sharply, but nobody edited its written rules. Before changing the definition, compare the people who entered and left, then trace each change to the evidence used at each evaluation. Separate ordinary time effects from changed customer circumstances, missing data and inconsistent interpretation.

A total-count comparison cannot do this. A small net decline can conceal substantial turnover, while a large decline may be entirely consistent with a rolling activity window. The useful question is not just “Why is the audience smaller?” It is “Which membership decisions changed, and does the evidence justify them?”

The audience segment drift investigation below uses a filled, fictional worksheet to show how to reach one of three decisions: accept the change, correct the evaluation, or renew review of the segment’s meaning.

1. Preserve the two versions before explaining the difference

Save the earlier and current membership lists before rerunning or editing anything. Label both with their evaluation time and time zone. Preserve the rule definition and the available source evidence behind each version, using only records the reviewer is authorised to access.

A useful comparison package contains:

  • A stable person or account identifier, at the same membership level in both versions.
  • The written rule and its operational interpretation, including date boundaries and treatment of missing values.
  • The source cutoff: which records were available when each version was evaluated.
  • The relevant activity dates, need status and exclusion evidence.
  • The membership decision and reason, where available.

If historical evidence was not retained, say so. A current record cannot reliably explain what the evaluator saw earlier. Mark reconstructed explanations as provisional rather than presenting them as observed history.

Also check that the counting unit stayed constant. A change from contact records to distinct people can look like audience drift even if nobody’s eligibility changed. Identity merges, splits and duplicate removal belong in a separate reconciliation category, not in a customer-behaviour explanation.

2. Measure turnover, not just the net change

Split the two lists into retained members, removals and additions. Retained members appear in both versions; removals appear only in the earlier version; additions appear only in the current version.

Fictional worked example: A business reviews an audience of people with recent interest in a service whose need remains open. Its hypothetical rule requires qualifying activity within the preceding 30 days, an explicitly open need, and no active exclusion. Missing required evidence produces a review hold rather than eligibility.

The hypothetical versions were evaluated at 09:00 UTC on 1 June and 8 June. Both use the same written rule and a rolling window that includes activity at the lower boundary and excludes activity after the evaluation time.

  • Earlier audience: 1,000 members.
  • Current audience: 910 members.
  • Retained: 790 members.
  • Removed: 210 members.
  • Added: 120 members.
  • Reconciliation: 1,000 − 210 + 120 = 910.

The hypothetical net decline is 90 members, or 9% of the earlier audience. But 330 people changed membership state. Reviewing only the net decline would hide much of the work needed to explain the change.

Reconcile these counts before interpreting causes. If they do not balance, check identifiers, duplicates and snapshot completeness first. An explanation built on incomparable lists will remain unreliable even if individual examples sound plausible.

3. Use a filled drift review worksheet

The following worksheet is entirely hypothetical. Each changed member has one primary reason so the counts reconcile. Secondary reasons are recorded separately: a person can have expired activity and a newly fulfilled need, but should not be counted twice in the headline totals.

Review header: Audience: recent interest with an open need. Comparison: 1 June to 8 June, 09:00 UTC. Written rule: unchanged. Initial release decision: hold unresolved membership changes from operational use pending review. Review owner: audience operations lead. Evidence owners: source-data steward and business process owner.

Removals: 210 hypothetical members

  • Expired activity — 100 removals. Earlier evidence: qualifying activity was inside the rolling window. Current evidence: the same activity is outside it, with no newer qualifying event available. Diagnostic action: recalculate the cutoff, inspect boundary cases and confirm which event types qualify. Provisional disposition: expected recency effect if the calculation matches the documented interpretation.
  • Newly fulfilled needs — 60 removals. Earlier evidence: need status was open. Current evidence: a dated completion record shows that the need was fulfilled between evaluations. Diagnostic action: verify that the completion concerns the same need, not an unrelated purchase or another person on the account. Provisional disposition: expected customer-state change if the evidence matches.
  • Missing source fields — 30 removals. Earlier evidence: an explicit open status was available. Current evidence: the required status field is blank, so the evaluator placed the record on review hold. Diagnostic action: compare historical and current evidence, determine whether the blank means unknown or an intentional status change, and ask the data owner to resolve it. Provisional disposition: unresolved data anomaly; do not describe these people as having completed their need.
  • Changed exclusion evidence — 20 removals. Earlier evidence: no active exclusion was recorded. Current evidence: an active exclusion is supported by a dated record. Diagnostic action: verify identity, scope and effective date. Provisional disposition: accept exclusions supported by evidence; retain unresolved records on hold rather than overriding the restriction.

Additions: 120 hypothetical members

  • New qualifying activity — 75 additions. Earlier evidence: no qualifying activity inside the window. Current evidence: a qualifying event occurred after the earlier evaluation. Diagnostic action: confirm the event type and occurrence time, rather than relying only on the time the record became available. Provisional disposition: expected entry if supported.
  • Reopened needs — 20 additions. Earlier evidence: the relevant need was fulfilled or closed. Current evidence: a dated record explicitly reopens that need, with qualifying activity still recent. Diagnostic action: distinguish a genuine reopening from a generic note or an unrelated request. Provisional disposition: expected entry only when the reopened need is the one the rule describes.
  • Restored source fields — 15 additions. Earlier evidence: required need status was unavailable. Current evidence: an explicit open status is now present. Diagnostic action: inspect the status history to distinguish newly supplied evidence from a real change in customer circumstances. Provisional disposition: potentially valid membership, but classify it as restored evidence rather than new demand.
  • Changed exclusion evidence — 10 additions. Earlier evidence: an active exclusion prevented membership. Current evidence: the exclusion is shown as inactive. Diagnostic action: require affirmative evidence that the exclusion ended; a blank field alone is insufficient. Provisional disposition: pending evidence review before inclusion.

Worksheet summary: The hypothetical primary reasons reconcile to 210 removals and 120 additions. The 30 removals with missing fields and 10 additions with changed exclusion evidence remain unresolved. Other categories are provisional until their diagnostic checks pass. No business-rule change is justified by the counts alone.

4. Separate time effects from changes in available evidence

A rolling window changes even when customers do nothing. That is different from a newly completed need, and different again from an old event becoming visible late. Keep three times distinct: when an event happened, when the source recorded it, and when the audience evaluation could use it.

For each removal attributed to recency, ask whether the last qualifying event genuinely aged out. Check the exact boundary, time zone and meaning of “day.” A calendar-day rule and a rolling-duration rule can disagree without either calculation being broken.

For additions, ask whether the activity is new or merely newly available. Late-arriving evidence may justify current membership while explaining no recent change in behaviour. That distinction matters when the team decides whether the segment still represents its intended audience.

Where preserved evidence and review tooling permit, use controlled comparisons:

  1. Keep the interpretation and source snapshot fixed; advance only the evaluation time. This isolates the effect of the clock on that snapshot.
  2. Keep the evaluation time and interpretation fixed; compare source snapshots, applying the same event-time cutoff. This exposes decisions affected by revised or newly available evidence.
  3. Keep time and evidence fixed; compare the earlier and current operational interpretations. Differences indicate evaluation or interpretation changes, not customer behaviour.

These comparisons are diagnostic aids, not necessarily additive explanations. Several conditions may change together. Record overlapping causes instead of forcing every counterfactual result into a single causal total.

5. Test whether the interpretation changed quietly

An unchanged sentence does not guarantee an unchanged decision. “Recent interest” may have started including passive activity. “Open need” may have shifted from an explicit status to the absence of a completion record. “No exclusion” may now treat an unknown value as a negative value.

Write down the operational answers to the ambiguous parts of the rule:

  • Which events count as qualifying activity, and which timestamp controls?
  • Does any completed need exclude the person, or only completion of the relevant need?
  • Is eligibility evaluated for a person, an account or a specific need?
  • What happens when evidence is missing, conflicting or undated?
  • Which evidence takes precedence when sources disagree?

Then build a small case set from the disputed patterns: a boundary-date event, an open need with an unrelated completion, a missing status, and conflicting exclusion evidence. Have reviewers apply the written definition independently before discussing the answers.

Disagreement is a reason to clarify the definition, not to choose whichever interpretation restores the old audience size. Retained members also deserve some inspection: stable membership can conceal an incorrect evaluation that happens to produce the same result.

6. Choose acceptance, correction or renewed review

Accept the drift when the evidence supports the changes and the interpretation remains consistent. In the hypothetical worksheet, verified expired activity and genuinely fulfilled needs are reasons to accept removals, not to widen the rule to recover volume.

Correct the evaluation or evidence when decisions diverge from the agreed definition. Examples include the wrong date boundary, missing evidence incorrectly treated as a negative answer, or an unrelated completion applied to the targeted need. Preserve the original comparison, document the correction and generate a corrected version for approval.

Renew business review when the evaluation is faithful but the audience no longer serves its intended purpose, or when reasonable reviewers cannot agree on the intended meaning. That review may produce a revised definition. Label it as a new version rather than presenting it as a repair to unchanged rules.

A mixed decision is often more useful than a blanket acceptance or rejection. The worksheet can accept verified recency effects while holding unresolved exclusion changes and assigning missing fields to the data owner.

7. Recover without making uncertain members eligible

Do not restore the entire earlier audience simply because it was previously approved. Some people may now have fulfilled their need or acquired a valid exclusion. Equally, do not treat the current version as correct merely because it is newer.

Isolate the affected reason groups. Keep missing values distinct from explicit negative values, and preserve current exclusion evidence while its meaning is reviewed. An unresolved membership decision should not become permission to contact someone.

Reviewing every changed record can be expensive. A reason-stratified sample can help test an initial diagnosis, especially when changes share a common pattern. Its limitation is that an exception may be missed. Record how much of each category was inspected, expand the review when exceptions appear, and avoid calling a sampled explanation exhaustive.

After a correction, rerun the reconciliation. Confirm that affected records changed as intended and that unaffected cases still behave consistently. If a replay is unavailable, use a documented manual case review and state the resulting evidence limitation.

8. Close the review with measurable evidence

The next action is to complete one reason ledger for the two preserved versions, not to rewrite the segment immediately. Assign an owner and a disposition to every changed-member category.

For the hypothetical worksheet, the immediate task is specific: resolve the 30 missing-status removals and verify the evidence behind the 10 exclusion-related additions. Report how many are confirmed, corrected or still held, then reconcile the final membership counts.

A review is ready for sign-off when:

  • The earlier count, removals, additions and current count balance.
  • Every changed member has a primary reason, including an explicit unresolved category where needed.
  • Verified explanations are distinguished from provisional ones.
  • Any interpretation change has a named owner and recorded approval.
  • Corrections have been checked against both affected and unaffected cases.

Keep the completed worksheet with the audience version. The next review can then compare changes against an established explanation rather than starting again with a surprising total.

Audience segmentationData qualityCRM operationsAudience review

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications