Include negative responses in campaign success decisions
Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.
A campaign can generate more useful replies and still be a poor candidate for wider use. If the same change also increases withdrawal requests, complaints or wrong-recipient replies, reporting only the positive response rate hides part of the result.
The practical answer is to make adoption conditional on two things: a meaningful improvement in the intended outcome and acceptable negative outcomes. Define both before launching the test, use comparable denominators and observation windows, and give unresolved classifications their own line in the report. A benefit threshold is not permission to ignore a guardrail breach.
The filled fictional scorecard below shows how this works. Its purpose is not to prescribe universal acceptable rates. It shows how a team can reach a defensible decision when the headline improves but the broader evidence does not.
1. Decide what counts as success before seeing the result
Start with a decision sentence, not a dashboard: We will expand this campaign only if useful replies improve enough to matter, negative outcomes stay within agreed boundaries, and the evidence is sufficiently complete.
That sentence separates three questions that are often collapsed into one:
- Benefit: Did the campaign produce more of the response the business actually wanted?
- Guardrails: Did it also create outcomes that make expansion unacceptable or require investigation?
- Evidence quality: Can the available data support the decision?
Define a useful reply in observable terms. For a fictional appointment campaign, it might mean that the intended recipient requests an appointment or gives information needed to assess an appointment request. A greeting, an automated acknowledgement or a request to stop messaging would not qualify merely because it is a reply.
Avoid a single weighted score that lets useful replies mathematically cancel complaints. Such a score can conceal the very trade-off the decision-maker needs to see. Keep benefit and guardrails visible, then state which conditions control expansion.
2. Classify negative outcomes without erasing mixed replies
Use a short classification guide that two reviewers could apply consistently. For this operating design, the relevant labels are:
- Withdrawal request: An explicit request to stop receiving the relevant communications. Count it even when the recipient is polite or also asks a business question.
- Complaint: An expressed objection about the outreach, its frequency, content or handling. Separate this from a neutral product question or an ordinary objection to the offer.
- Wrong recipient: A reply indicating that the contacted person is not the intended recipient or that the message relies on an incorrect identity assumption.
- Unknown classification: A response whose meaning remains unresolved after the initial review, including language ambiguity, incomplete context or reviewer disagreement.
Allow multiple labels on the same recipient. Someone can ask a useful question and then request no further campaign messages. Recording only the useful part would overstate success. Likewise, a complaint can also contain a withdrawal request.
For each metric, count a recipient once within the campaign variant, regardless of how many matching messages they send. Because labels overlap, their counts must not be added together to calculate total respondents. Maintain a separate unique-respondent count if the team needs one.
Do not interpret an unknown classification as neutral or favourable. It is a measurement limitation until resolved. Equally, no reply is not an unknown classification: it belongs in the campaign population but does not provide evidence of recipient sentiment.
3. Fix the denominator and observation window
For the worked example, every scorecard rate uses distinct recipients with at least one recorded delivered campaign message as its denominator. This measures responses among reached recipients. The team also records attempted recipients and delivery status separately so that differences in reach do not disappear.
This is one useful design, not the only valid denominator. Attempted recipients may be more appropriate for a business-level yield measure. What matters is that the report states its choice and does not compare useful replies per delivered recipient with complaints per respondent as though they were equivalent rates.
A respondent-only rate can be a helpful secondary view, but it can also mislead. If a variant changes how many people reply, the composition of the respondent group changes too. Keep the common campaign denominator in the primary decision scorecard.
Use the same follow-up window for both variants, measured from each recipient's delivery time. In the fictional example, the team waits seven days after the final delivery before closing the comparison. It separately monitors later negative responses and can reopen the decision if new evidence materially changes it.
Recipients should be assigned to comparable groups, ideally at random, with consistent eligibility, timing and follow-up treatment. If one variant goes to recent enquiries and the other to older contacts, the report cannot cleanly attribute the difference to the message.
4. Filled fictional decision scorecard
Fictional worked example: All counts, rates, time windows and thresholds in this example are hypothetical. They are not customer results, benchmarks or recommended universal limits.
A team tests a more direct appointment invitation against its current wording. The decision is whether to use the new wording for a wider campaign. Before launch, the campaign owner and a reviewer agree the rules below.
Test setup and evidence requirements
- Comparison: Current wording versus new wording, with recipients randomly assigned and no recipient in both groups.
- Attempted recipients: 2,100 in each group.
- Distinct delivered recipients: 2,000 in each group; the remaining 100 in each group are excluded from response-rate denominators and retained in the reach report.
- Observation window: Seven days after each recipient's delivery, with the decision held until all recipients complete that window.
- Counting rule: One count per recipient per label; labels may overlap.
- Evidence gate: Both groups must have at least 2,000 delivered recipients, complete follow-up and no more than 1.0% unknown classifications.
Benefit and guardrail results
- Useful replies: Current wording: 180 of 2,000, or 9.0%. New wording: 240 of 2,000, or 12.0%. Change: +3.0 percentage points. Pre-agreed benefit threshold: at least +2.0 percentage points. Result: benefit threshold met.
- Withdrawal requests: Current: 20, or 1.0%. New: 44, or 2.2%. Change: +1.2 percentage points. Pre-agreed review thresholds: new variant above 1.5%, or an increase above 0.3 percentage points. Result: both thresholds breached; no expansion pending review.
- Complaints: Current: 4, or 0.2%. New: 12, or 0.6%. Change: +0.4 percentage points. Pre-agreed review thresholds: new variant above 0.3%, or an increase above 0.1 percentage points. Result: both thresholds breached; no expansion pending review.
- Wrong-recipient replies: Current: 10, or 0.5%. New: 22, or 1.1%. Change: +0.6 percentage points. Pre-agreed review thresholds: new variant above 0.7%, or an increase above 0.2 percentage points. Result: both thresholds breached; investigate audience and identity assumptions.
- Unknown classifications: Current: 12, or 0.6%. New: 30, or 1.5%. Change: +0.9 percentage points. Pre-agreed evidence threshold: no more than 1.0% in either group. Result: new variant fails the evidence gate.
Recorded decision
Do not expand the new wording. It produced 60 additional useful replies, but also 24 additional withdrawal requests, eight additional complaints and 12 additional wrong-recipient replies. These are differences in recipient-level label counts, not counts of distinct additional affected people, because labels can overlap.
The useful-reply gain passes the benefit rule, but multiple negative-outcome guardrails fail and too many replies remain unclassified. The next step is a bounded investigation and, if warranted, a revised test—not a wider rollout.
5. Make thresholds meaningful rather than convenient
The example uses both an absolute ceiling and a permitted increase from the comparison group. They answer different questions. The ceiling asks whether the observed level is acceptable for this decision. The change threshold asks whether the new approach makes the outcome materially worse.
Using only a relative increase can exaggerate small movements from a low baseline. Using only an absolute ceiling can overlook a substantial deterioration that remains just below it. Showing counts, rates and percentage-point changes together makes those weaknesses easier to spot.
Choose thresholds according to the intended use, the consequences of getting the decision wrong and the team's ability to investigate. Do not select them after seeing which limits the preferred variant would pass. If a threshold proves poorly designed, document the change and evaluate it in a fresh decision cycle rather than retroactively declaring the original test successful.
Passing a threshold is also not proof that an observed improvement is reliable. Smaller counts can produce unstable rates. Pre-agree how uncertainty will be assessed, and treat an inconclusive comparison as inconclusive rather than lowering the benefit requirement to obtain a winner.
Finally, retain a qualitative escalation rule. A serious response should receive human review even if its category's aggregate rate is below the numerical limit. Numerical guardrails are not a reason to ignore individual signals.
6. Investigate the breach without rewriting the outcome
Start with the unresolved responses, then review the negative labels against the original conversations. Keep the original classification, the revised label, the reason for the change and the reviewer. This creates a visible correction trail instead of a silently improved dashboard.
For the fictional example, the investigation would ask:
- Did the new wording create useful engagement through clarity, or through confusion that required recipients to reply?
- Were withdrawal requests concentrated among recipients with more previous campaign exposure?
- Did a personalised phrase imply an identity or relationship that was incorrect?
- Were complaints about the message itself, cumulative contact frequency or something unrelated to the tested change?
- Did the new wording generate replies that the existing classification guide could not handle?
These questions suggest mechanisms; they do not establish causes. A wrong-recipient increase might reflect audience differences, clearer disclosure by recipients or misleading wording. Review assignment balance and message context before choosing a remedy.
Use segment analysis to locate a problem, not to search for a convenient winning subgroup. A promising subgroup discovered after the test is a hypothesis for another test. It is not automatically a basis for expanding the current variant.
7. Separate the adoption decision from recipient handling
Holding a rollout does not mean waiting to act on individual requests. Measurement, recipient handling and future campaign adoption are separate workstreams. A withdrawal request still needs appropriate handling while analysts decide how to classify the overall result.
For WhatsApp campaigns, recipient permission matters and opt-outs must be honoured; consult the current WhatsApp Business Messaging Policy for precise requirements. A favourable scorecard does not establish policy compliance, and these proposed guardrails are operating recommendations rather than platform mandates.
Record one of four adoption outcomes: expand, continue observation, revise and retest, or stop the variant. Assign an owner and a reason. In the fictional scorecard, the appropriate outcome is revise and retest only if the investigation produces a specific, testable correction; otherwise, stop the variant.
Do not label the current wording safe merely because it performed better on negative outcomes. It also needs to satisfy the team's own boundaries. A weak challenger does not validate the baseline.
8. Make the next report decision-ready
For the next campaign, replace the positive-only headline with a compact decision statement: Useful replies improved, but expansion is on hold because withdrawal, complaint and evidence-quality guardrails failed. Place the filled scorecard immediately beneath it.
Before launch, have the campaign owner and reviewer sign off on the useful-reply definition, negative labels, denominator, follow-up window, thresholds and action for each breach. Confirm that someone owns unresolved classifications and that the decision-maker can inspect the underlying evidence.
The measurable next action is simple: make the next adoption review include a completed value for every scorecard field, an explicit pass or breach against every agreed threshold, and a named decision owner. Where evidence is missing, record a hold—not an assumed pass.
Put the next conversation to work.
Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.
Explore Communications