Back to all articlesAI answer quality

Build an AI Answer Scorecard That Separates Serious Errors From Style Issues

Alactic Editorial10 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

A confident answer that invents a delivery date should fail review even if it sounds helpful. An accurate answer with an awkward sentence should not receive the same verdict. A single overall quality score makes those problems easy to confuse.

The practical fix is to evaluate five dimensions separately: grounding, completeness, uncertainty, relevance, and commitment accuracy. Give each dimension explicit scoring anchors, then apply release gates before considering style. Reviewers should record the claim or omission behind each deduction, not just their impression.

The scorecard below is a recommended operating design for response-level review. All business records, candidates, scores, thresholds, and pilot numbers are hypothetical. They are fictional worked examples, not Alactic customer results or evidence of a product capability.

1. Define the answer contract before reading the answer

Reviewers need a reference packet, not just generated text. Otherwise, fluent wording becomes the easiest available evidence of quality.

Assemble the customer request, relevant business records, their freshness, and the actions the assistant is authorised to perform. Then write an answer contract: the facts, limitations, and next steps a satisfactory response must cover. Prepare it before scoring to avoid adopting the candidate's framing.

Fictional worked example: A customer asks: “Has my replacement shipped? Will it arrive by Friday? Please send the tracking link and change the delivery address to the one in my previous message.”

The hypothetical reference packet establishes:

  • The replacement request is approved.
  • A replacement order is recorded as a draft. Dispatch is not recorded.
  • No tracking number or confirmed arrival date is available.
  • The requested address is visible in the conversation; the order still contains the old address.
  • The assistant can explain the records and prepare a request for a human operator. It cannot change the address or confirm dispatch.
  • No human review request has been submitted.

The contract requires the answer to distinguish approval from recorded dispatch, explain that Friday arrival is unconfirmed, identify unavailable tracking, state that the order address remains unchanged, and offer a permitted next step without claiming it happened.

Keep records and real-world events distinct. A draft order does not establish dispatch. Missing dispatch evidence also does not prove that physical dispatch never occurred. The supported statement is that dispatch is not recorded.

2. Score five dimensions with observable anchors

This hypothetical rubric uses 2 for meets the requirement, 1 for a bounded defect requiring revision, and 0 for a consequential failure. These are ordinal labels, not measurements with equal intervals.

Grounding: can material claims be supported?

  • 2: Every material factual claim has adequate support in the packet. Inferences are identified and justified.
  • 1: A non-material claim is unsupported or imprecise without changing the customer's understanding of status, options, or next steps.
  • 0: A material claim lacks support or conflicts with established evidence.

Claiming shipment when the packet contains only a draft order and no dispatch record merits zero. That claim is unsupported; it need not be proven false to fail grounding.

Completeness: does it cover the decision-relevant request?

  • 2: It addresses every material part of the request, including unavailable information and unresolved actions.
  • 1: It misses a secondary detail without preventing understanding of the situation or next step.
  • 0: It omits a central question or limitation needed for the customer's decision.

Completeness means covering what matters, not repeating every record. A fabricated answer to each question does not satisfy the contract.

Uncertainty: is confidence calibrated?

  • 2: It states established facts directly and makes unresolved matters explicit.
  • 1: It is mildly overconfident or unnecessarily hesitant without creating a material misunderstanding.
  • 0: It presents an unresolved outcome as certain or hides a critical unknown behind reassurance.

Do not reward indiscriminate hedging. Saying approval is merely possible understates a recorded approval.

Relevance: does the content help with this request?

  • 2: It focuses on the questions, necessary context, and useful next action.
  • 1: Removable material adds friction without burying the answer.
  • 0: It substantially answers a different question or lets unrelated content dominate.

Length alone does not determine relevance. Several customer questions may require several distinct explanations.

Commitment accuracy: are actions and promises represented truthfully?

  • 2: It distinguishes proposed, authorised, attempted, and confirmed actions. Promises stay within available authority and evidence.
  • 1: An action description is ambiguous but asserts no consequential completion or unsupported outcome.
  • 0: It claims an unperformed action, implies authority it lacks, or makes an unsupported material commitment.

This dimension catches an answer that describes the records correctly but falsely reports that a human review request has been submitted.

3. Make release gates stronger than the total

In this proposed rubric, any zero blocks release. Any one requires revision and re-review. Release readiness requires two in every substantive dimension, alongside any separate workflow checks.

Retain a total only as shorthand. Eight out of ten can still include a zero for falsely claiming an address change. Four strong dimensions cannot repair that failure.

Record style separately as clear, minor edit, or meaning obscured. Minor edits cover awkward phrasing, repetition, or tone preferences that do not change meaning. Meaning-obscuring wording requires revision; identify the substantive dimension affected.

One sentence can fail several dimensions. An invented arrival guarantee can be both ungrounded and an unsupported commitment. Record both effects, but assign corrective work to the single underlying defect rather than counting independent incidents.

4. Work through a fluent answer that fails

Hypothetical candidate A:

Your replacement has shipped and will arrive by Friday. I’ve updated your delivery address, and your tracking link will be sent shortly. Everything is taken care of.

This is concise, reassuring, and unsuitable for release.

  • Grounding — 0: Shipment is unsupported by the available records, which show a draft order and no recorded dispatch. Friday arrival is also unsupported.
  • Completeness — 0: It conceals unavailable tracking and the unresolved address change. Apparent coverage is not substantive completeness.
  • Uncertainty — 0: It turns unresolved dispatch and arrival into certainty.
  • Relevance — 2: It addresses the customer's topics. Relevance does not establish truth.
  • Commitment accuracy — 0: It falsely reports an address update and makes an unsupported promise about sending tracking.
  • Style — clear.
  • Decision — block release. Hypothetical total: two out of ten.

The repair is not simply softer wording. Saying the replacement should arrive Friday still introduces an unsupported expectation. Rebuild the answer from the available evidence and authority.

5. Compare caution with a complete, limited answer

Hypothetical candidate B:

I don’t have a tracking link or a confirmed arrival date. Please contact the support team for help.

This avoids invented shipping facts, but caution alone is insufficient.

  • Grounding — 2: The stated information gaps match the packet.
  • Completeness — 0: It omits approval, recorded draft status, and the unchanged order address. The referral does not explain the permitted next step.
  • Uncertainty — 2: It accurately identifies the unknowns it discusses.
  • Relevance — 2: It remains on topic.
  • Commitment accuracy — 2: It claims no unperformed action.
  • Style — clear.
  • Decision — block release. Hypothetical total: eight out of ten.

Hypothetical candidate C:

Your replacement is approved. The order is recorded as a draft, with no dispatch recorded. No tracking link is available, and arrival by Friday is not confirmed. The old address remains on the order; I have not changed it. I can prepare a request for a human operator to review the address change and shipping status. Would you like me to prepare that?
  • Grounding — 2: Material claims match the packet without claiming to know whether unrecorded physical dispatch occurred.
  • Completeness — 2: It addresses shipping status, Friday arrival, tracking, the address, and the next step.
  • Uncertainty — 2: It separates recorded approval from unresolved dispatch and arrival.
  • Relevance — 2: Each sentence supports the request.
  • Commitment accuracy — 2: It offers only the permitted preparation step and reports no unperformed action.
  • Style — minor edit: Some repetition could be removed without changing meaning.
  • Decision — substantively release-ready. Hypothetical total: ten out of ten.

Editing must preserve the distinction between preparing a request and submitting one. A smoother sentence that erases that boundary is not an improvement.

6. Reuse an evidence-first review record

Copy these fields for each answer. This filled hypothetical artifact records candidate A's address-change defect; its full dimension scores also capture the other failures.

  • Task: Explain replacement status and respond to the address-change request.
  • Reference: Fictional replacement packet above; no later updates supplied.
  • Answer contract: Approval versus recorded dispatch; unconfirmed Friday arrival; unavailable tracking; unchanged order address; permitted next step.
  • Scores: Grounding 0; completeness 0; uncertainty 0; relevance 2; commitment accuracy 0.
  • Candidate evidence: The sentence claiming the delivery address was updated.
  • Reference evidence: The old address remains on the order; the assistant cannot change it.
  • Root defect: Requested action represented as completed.
  • Consequence: The customer could believe no further address correction is needed.
  • Required repair: State that the order address remains unchanged and offer preparation of a human review request.
  • Decision: Block; re-review the complete replacement answer.
  • Evidence-gap owner: Not needed for this defect; the packet establishes the relevant state.

Require a specific claim or named omission for every deduction. If a reviewer cannot identify either, the deduction may reflect taste rather than an observable defect.

7. Handle evidence gaps without weakening the gates

When reviewers disagree, compare their packets and answer contracts before discussing scores. One may assume approval means dispatch; another may mistake preparing a request for submitting it. Do not average away different assumptions.

If the packet is stale, contradictory, or insufficient, mark the unresolved factual question as evidence-blocked. This describes the evidence problem, not an automatic verdict on every possible response.

Apply two distinct decisions:

  • An answer asserts the unresolved fact: Do not assign passing grounding on the assumption that it is probably correct. Mark the claim unscorable where truth cannot be established and block release pending evidence or revision. If the claim plainly exceeds available support, a grounding zero is appropriate.
  • An answer acknowledges the gap: Revise the answer contract to require a truthful limitation, any still-supported facts, and an authorised next step. Score that limited answer against the revised contract. It can pass without resolving or asserting the unknown fact.

For example, suppose two hypothetical records disagree about dispatch and neither can be established as current. A release-ready answer may explain that the records do not establish current shipping status and offer a permitted verification request. It must not select whichever record produces the most reassuring answer.

Record why the contract changed and assign the information gap to an owner. Contract revision must reflect unavailable evidence, not excuse an avoidable omission. The release gate remains intact: unresolved substantive scoring blocks release; an explicitly limited answer must still earn two in every dimension.

8. Calibrate the rubric before expanding review

Start with a hypothetical pilot of twelve appropriately handled requests spanning straightforward answers, missing information, multi-part questions, and action requests. Prepare each packet and contract, then have two reviewers score independently.

Compare agreement by dimension, release disagreements, and deductions without evidence. These describe the review process, not general model performance. Resolve each disputed anchor and rescore affected answers.

Watch for two failure patterns. Frequent zeros for harmless wording suggest overly broad anchors. Accepted answers with unresolved material claims suggest weak gates. Narrow the former without excusing the latter.

The pilot is complete when each blocked answer has a named defect and repair, each evidence gap has an owner or a documented limited contract, and each accepted answer passes every substantive dimension. Reviewers should be able to explain why fluent misinformation failed while an accurate but less elegant answer remained usable.

AI answer qualityQuality assuranceBusiness operationsHuman review

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications