Back to all articlesCustomer feedback

Build a qualitative feedback codebook that different reviewers can use

Alactic Editorial11 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

One reviewer labels a comment “onboarding.” Another calls it “usability.” A third chooses “support.” If all three labels are plausible, the problem may be the codebook rather than the reviewers.

To create a customer feedback codebook, define what text reviewers should code, give each code an operational meaning, specify what belongs elsewhere, and provide a route for unresolved cases. Then have reviewers independently code the same material before using the resulting themes to make decisions.

The goal is not to remove interpretation. It is to make interpretation visible, bounded, and easier to challenge. A useful codebook lets a reviewer explain a choice using a rule and a piece of evidence—not just “this felt like onboarding.”

1. Start with the decision the coding should inform

Write a short purpose statement before creating labels. It should identify the feedback population, the question being investigated, and what the results will not establish.

Fictional worked example: All comments, codes, counts, and pilot results below are hypothetical. A fictional business-software team wants to understand obstacles mentioned in feedback from customers setting up and using a shared workspace.

Purpose: Identify the obstacles customers explicitly describe during setup and routine use, so the team can choose issues for further investigation. The coding will not establish root causes, estimate prevalence among all customers, or determine the product roadmap by itself.

This purpose supports codes about reported obstacles. It does not require a code for every product area, emotional tone, or requested feature.

Define the material in scope as well: for example, feedback-form responses and selected support conversations from a specified period. Record how those records were selected. A collection dominated by support tickets answers a different question from a collection of unsolicited survey responses.

Keep topic codes separate from workflow fields. “Needs follow-up,” “reviewed,” and “urgent” describe handling decisions, not necessarily the subject of the feedback.

2. Decide exactly what reviewers will code

A coding unit is the piece of text to which a code applies. Without a shared unit, reviewers can disagree even when they understand the labels identically.

For this fictional starter codebook, use these rules:

  • Source record: One feedback response or one bounded customer conversation. Preserve its identifier and relevant context.
  • Coding unit: The smallest passage that expresses a complete issue or request. Keep an action and its stated consequence together.
  • Splitting: Split a passage when it contains separate issues that could be understood independently. Do not split merely at every sentence.
  • Context: Reviewers may read surrounding text to resolve references, but must record the passage supporting each code.
  • Multiple codes: Allow more than one code on a unit only when the passage independently supports each meaning. Do not apply several codes as a substitute for choosing.
  • Inference: Code what the customer reports. Do not infer a technical cause, intention, or missing capability from an unsuccessful attempt alone.

A hypothetical paraphrase such as “I could not find the invitation screen, and the help article used a different menu name” supports two distinct issues. Separate them if each passage remains meaningful. By contrast, keep “I could not find the invitation screen, so I asked a colleague” together: asking a colleague describes the consequence, not necessarily another obstacle.

Preserve the original text in the working dataset. Paraphrased examples belong in the codebook; they should not replace the evidence reviewers inspect.

3. Use this filled hypothetical starter codebook

The following entries form a deliberately narrow starting set. Each includes a stable identifier, definition, inclusion rule, exclusion, and boundary case. Every sample comment is a fictional paraphrase, not a customer quotation.

FIND — Difficulty locating an existing action or information

Definition: The customer reports not knowing where to find an action, setting, or piece of information.

Include: Searching menus, overlooking an entry point, or discovering an existing action only after assistance.

Sample paraphrase: I searched several screens before finding where to invite a teammate.

Exclude: Finding a control but not understanding its meaning belongs under MEAN. A reported failed action belongs under FAIL.

Boundary: “I cannot invite anyone” does not establish a findability problem. Ask for context or mark it unresolved.

MEAN — Unclear wording or instructions

Definition: The customer encounters wording but cannot understand its meaning or the action it requires.

Include: Confusing labels, unfamiliar terms, contradictory instructions, or an error message that does not explain what to change.

Sample paraphrase: I found the status setting, but I could not tell what its options meant.

Exclude: A missing location cue without a comprehension issue belongs under FIND. Purely aesthetic preferences are outside this codebook.

Boundary: If the customer says a misleading label caused them to overlook an action, apply FIND and MEAN only when both the search difficulty and wording problem are explicit.

FAIL — Reported action failure

Definition: The customer reports attempting an action that did not complete or produced an unexpected result.

Include: Reported errors, changes not saving, or a completed action apparently producing the wrong outcome.

Sample paraphrase: I selected save, but the previous value returned when I reopened the page.

Exclude: Do not treat this code as confirmation of a software defect. Searching for an action without attempting it belongs under FIND.

Boundary: An error message may support FAIL and MEAN if the customer explicitly reports both a failed action and incomprehensible recovery instructions.

EFFORT — Excess work in a functioning process

Definition: The customer describes a process that works but requires unwanted repetition, manual transfer, or avoidable steps.

Include: Re-entering information, repeating the same action for multiple records, or maintaining a manual workaround.

Sample paraphrase: Updating the records works, but I have to repeat the same change separately for each one.

Exclude: Extra attempts caused solely by a reported failed action belong under FAIL. A capability request without described effort belongs under REQUEST.

Boundary: A request for a bulk action can receive EFFORT and REQUEST when the customer explicitly describes repetitive work and asks for that action.

HELP — Difficulty obtaining useful assistance

Definition: The customer reports an obstacle in getting relevant assistance after seeking help.

Include: An unanswered request, irrelevant guidance, or being passed between people without resolution.

Sample paraphrase: I asked how to change the setting, but the reply explained a different task.

Exclude: Merely contacting support is not evidence of a support problem. Confusing help content belongs under MEAN when the issue is its wording.

Boundary: Code an underlying product obstacle separately only if the passage describes it. Do not infer it from the fact that help was requested.

REQUEST — Explicit request for a capability or change

Definition: The customer directly asks for a new or changed capability, behavior, or option.

Include: Suggestions framed as desired changes, including requests that may already be possible.

Sample paraphrase: I would like an option to apply this change to several records together.

Exclude: Complaints without an explicit desired change. Do not translate every obstacle into an implied feature request.

Boundary: This code records a request, not verified product absence. Use a separate verification field if someone later checks whether the capability exists.

Administrative dispositions: Use UNRESOLVED for potentially relevant material whose meaning cannot yet be determined, and OUTSIDE for material outside the stated question. Neither is a customer theme. Record a reason rather than allowing either to become a miscellaneous bucket.

4. Make ambiguity a documented route, not a private judgment

Give reviewers a short decision sequence:

  1. Identify the words supporting the proposed code.
  2. Check its inclusion and exclusion rules.
  3. If another code seems plausible, compare the boundary rules rather than choosing the broader label.
  4. If the text independently supports both codes, apply both and retain the evidence for each.
  5. If the meaning remains unclear, mark UNRESOLVED and state what information or rule is missing.

For the hypothetical paraphrase “Setup was a nightmare,” the correct response is not to guess between FIND, MEAN, FAIL, and EFFORT. The comment expresses dissatisfaction but does not identify an obstacle under these definitions. Keep it unresolved unless surrounding context supplies the missing detail.

Maintain an ambiguity log with the record identifier, passage, candidate codes, reviewer reasoning, decision, and any codebook change. Name a person responsible for resolving disputes and a review cadence. Resolution may be “insufficient evidence,” not necessarily a thematic code.

Separate disagreements about meaning from disagreements about segmentation. If reviewers selected different passages, settle the unit boundary first; otherwise, the code debate may be comparing different evidence.

5. Pilot independently before discussing answers

Choose a small batch containing both ordinary feedback and likely boundary cases. Include short comments, multi-issue passages, requests, and vague statements. This is a diagnostic sample, not a basis for estimating how common each issue is.

In the hypothetical pilot, two reviewers first segment the same 12 records independently. They reconcile unit boundaries using the written rules, producing 20 shared units. They then independently code those units without seeing each other’s labels.

This staged design separates segmentation problems from code-selection problems. Its trade-off is that it does not test the entire workflow independently from start to finish. A later fresh-record check should include both.

Compare exact code sets for each shared unit, including dispositions. Also inspect which codes are confused, where reviewers add unsupported secondary codes, and which passages repeatedly become unresolved.

Do not report only agreement after discussion. Consensus is useful for producing a final dataset, but it hides the ambiguity the independent pilot was intended to expose.

6. Document a revision, then test it on fresh material

The fictional pilot produces exact code-set agreement on 14 of 20 units. The six disagreements reveal specific rule problems: three concern FAIL versus EFFORT, two concern HELP, and one concerns an inferred REQUEST.

Hypothetical revision log, version 0.1 to version 0.2:

  • Observed problem: Reviewers used EFFORT whenever a failed action required repeated attempts.
  • Previous wording: EFFORT covered activities requiring extra work, without distinguishing a functioning process from retries after failure.
  • Revised rule: Use EFFORT for burdensome but functioning processes. Use FAIL for reported unsuccessful actions. Add EFFORT only when a separate recurring burden is explicitly described.
  • Boundary example added: A fictional paraphrase about pressing save repeatedly because a change disappears receives FAIL, not automatically EFFORT.
  • Other clarification: Contacting support alone does not justify HELP; describing a problem alone does not justify REQUEST.
  • Dataset action: Revisit all pilot units where the affected codes were used or considered, not just the six disagreements.

On a fresh hypothetical batch of 20 shared units, reviewers agree on 18 exact code sets before discussion. This is a local diagnostic result, not a benchmark or proof of reliability across other datasets. Inspect the remaining disagreements and check whether reviewers are still identifying units consistently on fresh records.

Set a readiness rule before viewing results. A practical rule might require no unresolved definition disputes, evidence for every applied code, and acceptable performance on the boundary cases that matter to the intended decision. Do not let a high overall agreement figure conceal confusion around a rare but consequential issue.

7. Release a version reviewers can actually maintain

Publish the purpose, unit rules, code entries, ambiguity route, owner, version date, and revision log together. Store the codebook version beside each coding batch.

During production, keep a shared review sample rather than sending every disagreement to informal chat. If a new pattern appears, log it as a candidate code with supporting passages and an explanation of why existing codes fail. Avoid creating a permanent code for every unusual sentence.

When a definition changes, identify which earlier records might be affected and whether they need recoding. Counts produced under materially different definitions should not be silently combined.

Keep codes distinct from later themes. A theme may connect several codes—for example, FIND and MEAN might contribute to an interpretation about setup guidance. Preserve the underlying coded passages so readers can inspect that interpretation rather than accepting a theme name as evidence.

8. Define the denominator before using the results

Decide whether a count represents coded passages, unique feedback records, or unique customers. These answer different questions. A long conversation can contain several passages with the same code and overwhelm a passage-level count.

For a record-level summary, count each code once per record and state that records may receive multiple codes. Report unresolved material separately. Do not describe a selected feedback sample as a percentage of all customers.

Your next action is to schedule one calibration session: draft the purpose statement, adapt these entries, select a mixed pilot batch, and assign independent reviewers. Leave with a versioned codebook, a disagreement log, and a decision about what must change before broader coding begins.

The useful outcome is not simply more agreement. It is being able to explain why a passage received a code, where interpretation remains uncertain, and which evidence supports the decisions that follow.

Customer feedbackQualitative researchCodebooksReview workflows

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications