Compare Two AI Answer Versions Without Rewarding Verbosity
Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.
A revised AI answer adds three paragraphs, sounds more certain, and wins the team’s vote. But did it help the reader make a better decision—or simply make the same advice look more substantial?
To compare AI answer quality, give both versions the same reader task and evidence. Check their consequential claims, then judge whether the reader can complete the task correctly. Evaluate factual support, task coverage, and uncertainty before considering polish. More words should earn credit only when they add something necessary.
The filled worksheet below demonstrates this approach with a fictional support scenario. It also shows why the right decision can be “choose this version for revision,” rather than “publish the winner.”
1. Fix the task before reading either answer
A comparison becomes unreliable when reviewers decide what matters after reading the responses. A confident explanation can quietly redefine the task: instead of helping an agent identify the next action, it impresses reviewers with background detail.
Write a short evaluation contract first:
- Reader: Who will use the answer, and what do they already know?
- Task: What decision or action must the answer support?
- Evidence boundary: Which documents, records, and assumptions may it use?
- Required coverage: Which questions must it answer?
- Release blockers: Which errors would make it unsuitable for use?
- Presentation: Does the reader need a checklist, a draft reply, or a recommendation?
Describe success as observable behavior. “Helpful and professional” is difficult to adjudicate. “Identifies missing information, gives the correct handoff, and avoids promising an unconfirmed outcome” gives reviewers something to inspect.
Separate response comparison from explaining why a system changed. You can compare existing outputs against a common task. But if the prompt, evidence, and generation settings also changed, the pair cannot isolate which change caused the difference.
2. Freeze a small, shared evidence packet
Reviewers need a common basis for checking claims. Otherwise, one person supplies remembered policy while another judges only what appears in the answer.
Use the smallest packet that supports the task: relevant operating instructions, the customer’s message, and necessary case facts. Record which version of each item was used. Keep the packet unchanged during the comparison.
Distinguish established facts, permitted inferences, and unknowns. A permitted inference must follow from supplied facts; it cannot depend on an unstated assumption. A missing fact is not permission to insert a familiar business practice.
If the packet does not specify a completion deadline, neither answer should receive credit for adding one. If essential evidence is unavailable, pause the comparison or narrow the task. You can still compare which answer recognizes the gap, but you cannot establish an unavailable substantive outcome.
3. Use this filled hypothetical comparison worksheet
Fictional worked example: Every business rule, case detail, quantity, and answer below is hypothetical. These are not Alactic policies, product capabilities, or customer results. Reuse the field structure for another comparison, replacing the task, evidence, answers, and findings.
Task and evidence
Reader: An internal support agent at a fictional subscription business.
Task: Determine the next step for a possible duplicate charge and prepare a customer-facing reply.
Customer message: “I see two charges for my renewal. Can you refund one? When will this be fixed?”
Shared evidence packet:
- E1: A possible duplicate-charge case requires the invoice reference and the dates and amounts of both reported charges.
- E2: If any required details are missing, the agent should request them before routing the case.
- E3: Once the required details are available, the agent routes the case to billing review. Billing review determines whether there is a duplicate charge and whether a refund is appropriate.
- E4: The agent must not confirm a refund before billing review. The packet contains no review or refund completion timeframe.
- E5: The current case has an invoice reference and both reported amounts, but no charge dates. It contains no billing-review decision.
Required coverage: Identify the missing dates; explain the conditional handoff; avoid confirming a duplicate or refund; address timing honestly; supply a usable customer reply.
Release blockers: Invented approval, unsupported completion deadline, or routing that skips the missing-information step.
Answer versions
Version A:
This looks like a duplicate renewal charge. Because the invoice reference and matching amounts are already available, send the case directly to billing and let the customer know that one payment will be refunded within five business days. Ask for the charge dates as an optional follow-up so the record is complete.
Suggested reply: “Thanks for flagging this, and sorry for the inconvenience. We’ve identified a duplicate renewal payment and will refund the extra charge within five business days. Our billing team will take care of the correction. If convenient, please send the dates of the two charges so we can complete our records. There’s no further action needed from you while we resolve this.”
Version B:
Ask for the dates of both charges before routing the case. Once those dates are available, send the case to billing review to determine whether there is a duplicate and whether a refund is appropriate. A refund is not confirmed.
Suggested reply: “Thanks for flagging this. Please send the dates of both charges. Once we have them, we can send the case to billing review to check whether a refund is appropriate.”
Claim checks
- A: The reported amounts match. Unsupported. E5 establishes that both amounts are available, not that they are equal. Even equal amounts would not, by themselves, establish a duplicate charge.
- A: The case can go directly to billing. Contradicted by E2 and E5. The missing dates are a prerequisite, not optional recordkeeping.
- A: A duplicate payment has been identified. Unsupported. E3 assigns that determination to billing review; E5 records no decision.
- A: A refund will arrive within five business days. Unsupported on both approval and timing. E4 prohibits confirmation and supplies no timeframe.
- A: No further customer action is needed. Contradicted by the missing dates in E5. It also conflicts with A’s request for those dates.
- B: Request both charge dates before routing. Supported by E1, E2, and E5.
- B: Billing review determines whether there is a duplicate and whether a refund is appropriate. Supported by E3.
- B: A refund is not confirmed. Supported by E4 and E5.
Coverage and uncertainty
- Missing information: A makes the dates optional. B correctly treats them as required.
- Handoff: A gives the wrong sequence. B gives the supported conditional sequence.
- Outcome: A converts a reported problem into a confirmed duplicate and refund. B preserves the unresolved status.
- Timing: A invents a deadline. B leaves the customer’s question unanswered.
- Customer reply: A’s reply contains unsupported commitments. B’s reply needs an explicit timing limitation.
Uncertainty finding: A sounds decisive by removing uncertainty the evidence still contains. B separates the next action from the unresolved outcome, but does not explain that no timeframe is available.
Selection, repair, and recheck
Disposition: Choose B as the revision base, not as a release-ready answer. It preserves the required process without unsupported commitments.
Required repair: Add a direct response to the timing question without inventing a deadline.
Repaired customer reply: “Thanks for flagging this. Please send the dates of both charges. Once we have them, we can send the case to billing review to check whether there is a duplicate charge and whether a refund is appropriate. I can’t confirm a completion date from the information available.”
Recheck result: The repaired reply requests the missing dates before routing, leaves the outcome to billing review, and addresses timing through an explicit limitation. It makes no claim that the amounts match. Against this hypothetical packet, it covers the required customer-facing content without adding an approval or deadline.
4. Check consequential claims, not sentence counts
A single sentence can contain several claims: an eligibility decision, a promised action, a deadline, and an assertion about responsibility. Split those apart to expose hidden errors.
Check statements that could change the reader’s action, expectation, cost, access, or understanding of the outcome. Record the evidence and classify each as supported, contradicted, unsupported, or appropriately conditional.
The word “matching” in Version A illustrates why small modifiers matter. It adds a case fact that the packet never supplied. Removing it would correct that invention, but would not repair the premature duplicate determination, handoff, or refund promise.
Avoid turning raw claim totals into a quality score. A long answer can accumulate low-value truths while burying one serious error. A short answer can avoid falsehoods by saying almost nothing. Neither should win through arithmetic alone.
Also inspect internal consistency. Requesting information while saying no further action is needed creates confusion even before reviewers consult the evidence packet.
5. Judge factual usefulness before presentation
Use a decision order rather than an undifferentiated score:
- Check release blockers. Polish cannot compensate for a material unsupported claim.
- Check coverage. Does the answer enable the required decision or action?
- Check uncertainty. Does it distinguish facts, conditional steps, and unresolved outcomes?
- Compare usability. Can the intended reader find and apply the answer?
- Consider style. Among substantively acceptable answers, prefer the presentation that fits the reader.
This does not mean shorter always wins. Additional explanation earns its place when it resolves ambiguity, supplies a necessary condition, or prevents a likely mistake. It is overhead when it repeats the conclusion or adds irrelevant background.
Ask: “What task-relevant information disappears if this passage is removed?” If only rhetorical emphasis disappears, do not count the passage as a factual improvement.
Appropriate uncertainty must not become evasiveness. B’s missing timing response is a real coverage defect. The repair is an explicit limitation, not a fabricated date.
6. Make reviewer disagreement traceable
Hide model names and revision history where practical. Label outputs neutrally, vary display order across reviewers, and ask reviewers to assess them independently before discussion. These practices reduce opportunities for preference to dominate; they do not eliminate judgment.
Ask each reviewer for a verdict, decisive evidence references, and release blockers. “A feels more complete” needs a follow-up: which requirement does A satisfy better, and is its content supported?
Classify disagreements before settling them:
- Evidence: Reviewers interpret the packet differently. Clarify it or mark the issue unresolved.
- Task: Reviewers expect different outputs. Repair the evaluation contract.
- Severity: Reviewers agree on the defect but differ on its consequence. Identify the wrong action it could cause.
- Style: Both answers meet the substantive bar. Record a preference without calling it a factual gain.
Allow “tie,” “neither acceptable,” and “insufficient evidence.” A forced winner can conceal two bad answers or turn an evidence gap into false certainty.
7. Repair regressions without blending blindly
A revision may improve one dimension while damaging another. Record gains, losses, and unchanged elements separately. An answer might improve the handoff sequence but omit a requested customer reply.
Do not automatically combine preferred passages. A sentence that works in one context can contradict another. Copying A’s reassuring conclusion into B would restore the unsupported refund promise.
Instead, identify the unmet requirement, make the smallest adequate edit, and rerun claim and coverage checks on the complete result. The worksheet demonstrates that cycle: select B, repair timing coverage, then inspect the full reply.
If both answers fail, reject both while retaining the findings. If the packet changes, reopen dependent claims. Route unresolved material questions to the evidence owner rather than voting on what is probably true.
8. Keep a decision record others can reproduce
Save the task contract, frozen evidence packet, both outputs, claim checks, coverage findings, and final disposition together. The worksheet’s headings provide a reusable record structure; replace the content rather than carrying fictional rules into a real case.
Track observable findings separately: unresolved consequential claims, unmet requirements, release blockers, and whether the repaired answer passes recheck. A pairwise preference is not a general benchmark or proof that future answers will improve.
The immediate goal is a selection rationale another reviewer can reproduce from the same evidence. If it depends on “sounds smarter” or “looks more thorough,” the comparison is not finished.
Put the next conversation to work.
Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.
Explore Communications