Find Repeated Answer Defects Without Counting Every Mistake Equally
Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.
Your review queue contains missing caveats, confident promises, invented details and refusals that should never have happened. Counting each flagged sentence as one mistake will produce a ranking, but not necessarily a useful one. A frequent, obvious refusal can outrank a rarer fabricated detail that a reader is likely to trust.
To prioritise AI answer quality problems, group findings by the underlying answer failure, measure recurrence against relevant opportunities, and assess both the consequence of acting on the answer and the likelihood that someone will catch the defect first. Then make an explicit decision about which family needs attention now, which needs more evidence and which can wait.
The worksheet below shows how to do that without hiding the judgement inside a single numerical score. It concerns interpretation of review findings—not system architecture, source repair or a general performance dashboard.
1. Make the reviewed answer—not every flagged phrase—the counting unit
Raw error totals often reflect annotation habits as much as answer quality. One reviewer may flag an unsupported delivery promise once. Another may mark its three appearances separately. If those annotations become three independent incidents, repetition within an answer gets confused with recurrence across answers.
Keep the original annotations, but create an answer-level record with:
- A stable answer identifier and the task the user was trying to complete.
- The evidence available to the reviewer when assessing correctness.
- Each distinct defect family present in the answer.
- The decision or action the answer might influence.
- Whether a normal reader could recognise the problem without specialist checking.
Count an answer once within each defect family, even if the same failure appears repeatedly. An answer may belong to more than one family, so family counts must not be added together and presented as the number of defective answers.
Also keep the sampling method visible. A collection assembled from complaints can reveal important failures, but its defect frequencies do not describe all answers. Label it as a complaint sample rather than presenting its percentages as an overall quality rate.
2. Group by the answer failure, not just the topic
Topic labels are useful for finding examples, but weak for deciding what to improve. A group labelled “campaign questions” could contain invented product features, missing conditions and unnecessary refusals. Those are different problems even when they occur in the same conversation.
Use a small set of behavioural definitions with clear boundaries:
- Unsupported promise: the answer presents a future outcome as assured without sufficient support. The defect is unwarranted certainty, not merely an incorrect detail.
- Omitted condition: the answer gives otherwise relevant guidance but leaves out a condition that changes whether it applies or what the user should do.
- Fabricated detail: the answer supplies a specific fact, capability, status or value that the available evidence does not support.
- Unnecessary refusal: the answer declines a permitted, answerable request despite having enough information to provide useful help within the task boundary.
These are observable failure modes, not proven technical causes. A fabricated detail might result from several different mechanisms. Record a proposed explanation as a hypothesis until further evidence supports it.
Allow multiple labels when they represent distinct failures. In a hypothetical answer, an invented completion status followed by an assurance of success could justify both fabricated-detail and unsupported-promise labels. Do not add a second label merely because it makes the same problem sound more serious.
When a family remains too broad to support a decision, split it by consequence or task. Missing an optional explanatory note is not equivalent to omitting a prerequisite for taking action.
3. Compare recurrence, consequence and detectability separately
Use three separate judgements before deciding priority. Keeping them visible makes disagreements easier to resolve than arguing about an unexplained total score.
Recurrence: how often does the failure appear when it could occur?
Record both the number of affected answers and the number of relevant opportunities. A promise defect should be assessed against answers addressing outcomes, not automatically against every answer in the collection.
Define opportunities before looking for defects where possible. Otherwise, reviewers may include only the cases that make a family look unusually good or bad. Preserve the task mix as well: a targeted sample of difficult questions should not be compared directly with a broad sample of routine ones.
Consequence: what could change if the reader trusted it?
Describe a plausible action, not just a severity adjective. Could the user repeat a task unnecessarily, stop seeking help, communicate an unsupported commitment or proceed on a false status? Distinguish an observed downstream outcome from a consequence that remains hypothetical.
Consider reversibility and reach. An error affecting one easily corrected choice differs from one that could be reused in a team instruction. Do not assume wider exposure without evidence.
Detectability: who would notice before acting?
A defect that is obvious to a reviewer with reference material may be difficult for the intended reader to spot. Assess detection in the normal reading context, not under ideal review conditions.
Use concrete descriptions: visible refusal, discoverable through a routine reference check, or plausible detail requiring specialist verification. Poor detectability raises concern; it does not prove that harm occurred.
4. Filled hypothetical defect-analysis worksheet
Fictional worked example: Every answer, count, percentage, consequence and priority decision in this worksheet is hypothetical. These are not customer results or benchmark findings.
Review frame: A team examines a hypothetical collection of 120 assistant answers covering product information, process explanations and status questions. Reviewers mark all relevant opportunity categories for each answer before assessing defects. Categories overlap. They count affected answers once per family, retain the supporting evidence and separately note disputed cases. Consequences below are plausible risks, not observed outcomes.
Worksheet entry A: unsupported promises
- Observed failure: answers turn a contingent outcome into an assurance, such as claiming that a requested result will definitely occur.
- Recurrence: 6 affected answers among 24 outcome-related opportunities: 25% within this hypothetical sample.
- Consequence: a user could make a commitment to someone else or stop checking progress because success appears assured.
- Detectability: poor for a reader who cannot independently inspect the conditions governing the outcome. Confident wording provides no visible warning.
- Uncertainty: the sample does not establish whether the same behaviour occurs outside outcome questions.
- Priority decision: second. Address assurance language and the distinction between known status and possible outcome after the fabricated-status cases below.
- Next review criterion: outcome answers state what is known, identify material dependencies and avoid implying certainty that the evidence cannot support.
Worksheet entry B: omitted conditions
- Observed failure: answers explain a process but leave out a prerequisite that affects whether the guidance applies.
- Recurrence: 12 affected answers among 40 conditional-guidance opportunities: 30% within this hypothetical sample.
- Consequence: users could attempt an unsuitable next step or need to repeat work after discovering the missing prerequisite.
- Detectability: mixed. Some omissions become visible during the task; others require knowledge the user was asking the assistant to provide.
- Uncertainty: reviewers need to distinguish missing essential conditions from optional detail. Borderline cases remain separately marked rather than automatically counted as failures.
- Priority decision: third as a family, with essential-condition omissions escalated individually if their consequences match the highest-priority cases.
- Next review criterion: guidance includes conditions that change the recommended action, without becoming a catalogue of every possible exception.
Worksheet entry C: fabricated details
- Observed failure: answers invent a completion status or product capability rather than acknowledging that the available evidence does not establish it.
- Recurrence: 5 affected answers among 30 factual-detail opportunities: approximately 17% within this hypothetical sample.
- Consequence: a user could stop following up on unfinished work or plan a task around a capability that has not been established.
- Detectability: poor. The statements look like ordinary factual answers and require an authoritative check to challenge.
- Uncertainty: this family contains two subtypes. Reviewers retain separate completion-status and capability labels so that improvement in one cannot conceal failure in the other.
- Priority decision: first, starting with invented completion statuses. Lower observed recurrence does not outweigh the combination of decision-changing content and weak detectability.
- Next review criterion: specific claims are supported by the available evidence; unknown status remains unknown rather than being filled in.
Worksheet entry D: unnecessary refusals
- Observed failure: answers decline straightforward explanatory requests that are within the permitted task and supported by the available material.
- Recurrence: 9 affected answers among 26 clearly answerable requests: approximately 35% within this hypothetical sample.
- Consequence: users lose time, abandon a useful interaction or seek help elsewhere. No wider downstream effect is established in this example.
- Detectability: high. The user can see that the answer has refused to help, although they may not know the refusal was unnecessary.
- Uncertainty: reviewers must confirm answerability rather than treating every disappointing refusal as defective.
- Priority decision: fourth, despite having the highest within-family percentage. Revisit sooner if evidence shows that these refusals block a consequential or time-sensitive task.
- Next review criterion: the answer provides permitted help, uses a narrow boundary where needed and does not refuse the entire request unnecessarily.
5. Turn the worksheet into a defensible decision
The hypothetical ranking is not “rare errors matter more.” It is a judgement that invented statuses and unsupported assurances can misdirect action while giving the reader little reason to question them. Refusals are more frequent in their opportunity set, but their immediate failure is more visible.
A useful decision statement names the trade-off: We are addressing fabricated completion statuses first because they can cause users to stop following up, and ordinary readers cannot readily detect them. We accept a temporary backlog of visible, lower-consequence refusals while checking that none block critical tasks.
Avoid multiplying ordinal labels into a precise-looking risk number. Calling consequence “high” and detectability “low” does not create reliable arithmetic. If a team uses numerical scoring for sorting, preserve the underlying evidence and allow explicit overrides.
Set an escalation rule as well. A single credible example with serious consequences may deserve immediate attention before recurrence is established. That is a precautionary decision, not evidence that the defect is widespread.
6. Recover when the classification or priority is wrong
Expect the first worksheet to change. The useful question is whether new evidence changes the decision, not whether the original ranking can be defended indefinitely.
- Reviewers disagree about the family: revisit the answer, task and evidence together. Clarify the boundary with an inclusion and exclusion example, then reassess affected records.
- The denominator is unclear: keep the defect count, mark recurrence as unresolved and review opportunity eligibility. Do not manufacture a percentage.
- A broad family hides different consequences: split it and re-rank the subtypes. Do not let harmless missing detail dilute essential-condition omissions.
- Only complaints were reviewed: retain the findings as evidence of failure modes, then examine a separately selected sample before making prevalence claims.
- A change reduces one defect but increases another: assess the exchange directly. Fewer fabricated answers achieved through blanket refusals is not automatically an improvement.
Do not erase uncertainty to make the worksheet look finished. A provisional priority with a named evidence gap is more actionable than a confident ranking built on inconsistent labels.
7. Make the next review test the decision
Finish with a bounded review assignment, not an instruction to “improve accuracy.” Select one defect subtype, name an owner and define what evidence would show progress or trigger reconsideration.
For a hypothetical next review, the team could examine 20 new status questions, including answerable cases and cases where status is unavailable. It would record fabricated-status answers, appropriate uncertainty, useful supported answers and unnecessary refusals. This is a suggested review design, not a statistically conclusive test.
Keep the earlier examples as a separate comparison set. Correcting familiar cases is useful, but it does not establish that the failure has stopped recurring in new answers.
The measurable next action is to complete one worksheet from the existing review collection: count affected answers, establish eligible opportunities, describe the likely consequence, assess reader detectability and write the priority decision. At the next review, compare those same fields—not just the total number of red marks.
Put the next conversation to work.
Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.
Explore Communications