Back to all articlesAI knowledge preparation

Replace an Approved Knowledge Source Without Leaving Two Active Rules

Alactic Editorial10 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

A revised service coverage guide has been approved and added to the assistant’s knowledge collection. Yet the previous guide is still retrievable. Ask whether a certain repair is covered, and the assistant can now find two approved passages giving different answers.

To replace an outdated AI knowledge source, treat publication and retirement as one controlled change. Identify the passages being superseded, assign an exact effective time, link the old and new authorities, remove the predecessor from current-answer retrieval, and test both retrieval results and generated answers. Uploading the replacement alone is not enough.

The central control is a supersession record: a short, owned account of what changes, when it changes, and what evidence is required before the replacement is considered complete. The fictional worked example below shows how to build one for a revised service coverage guide.

1. Define what stops being authoritative

Start with the business rule, not the file operation. A replacement might retire an entire guide, only certain sections, or one regional appendix. Those are different changes and need different boundaries.

Record whether the new source fully replaces the predecessor or amends selected passages. For a partial amendment, identify how the remaining content stays authoritative. Leaving an old guide active with an informal instruction to ignore selected paragraphs creates a dependency on the assistant interpreting that instruction correctly every time.

Where practical, publish a consolidated current guide rather than asking retrieval to reconcile a base document and several amendments. The trade-off is extra editorial work, but the resulting authority is easier to inspect.

  • Full replacement: the predecessor is excluded from current-answer retrieval after the cutover.
  • Partial replacement: obsolete passages are excluded, and retained passages have an explicit current status and source relationship.
  • Historical retention: the predecessor remains available only through a deliberately separate historical route, if that use is required.

Do not equate archiving with deleting. A document may need to remain in a records repository while being unavailable to an assistant answering current service questions. Conversely, deleting the original file does not demonstrate that extracted passages, indexed copies, or cached answers have disappeared.

2. Map passages and copies before setting the cutover

Build a small impact map of the changed rules. For each one, record the old passage, its replacement, the expected answer change, and any related content that repeats the rule. Use stable document identifiers and section references where available; filenames alone are weak identifiers because people can rename files or upload duplicates.

Look beyond the source folder. The same rule may appear in an extracted text collection, a help article, an agent reference sheet, or a manually maintained answer. These are candidates to inspect, not assumptions about how any particular system works.

For each location, establish how retirement is performed and how its result can be observed. Does removing a file also remove its indexed passages? Does a rebuild need to finish? Is there a separate answer cache? If the answer is unknown, record that uncertainty as a cutover dependency rather than assuming synchronization.

Also inspect mixed passages. A retrieved chunk may contain both a superseded coverage rule and an unchanged contact instruction. Keeping that chunk for the contact instruction can preserve the obsolete rule. Rebuild or replace the passage so that valid content does not carry invalid content back into retrieval.

3. Fill a supersession record before changing the live source set

Fictional worked example: every organization, identifier, date, coverage rule, role assignment, and test outcome in this example is hypothetical. The record illustrates a recommended operating design, not an existing product control.

Filled hypothetical supersession record

  • Change ID: SUP-SERVICE-042.
  • Change owner: Service Operations Lead.
  • Content approver: Service Coverage Manager.
  • Retrieval operator: Knowledge Administrator.
  • Verification owner: Support Quality Lead, separate from the person performing the replacement.
  • Predecessor: Service Coverage Guide, revision C; document ID SCG-C; source record SRC-118.
  • Replacement: Service Coverage Guide, revision D; document ID SCG-D; source record SRC-164.
  • Scope: full replacement for current service enquiries. Revision C is retained in the historical records repository, outside the current-answer collection.
  • Business effective time: 14 September 2026 at 09:00 UTC. Revision C applies before that time; revision D applies at and after it.
  • Replacement relationships: SCG-C has status retired and replacement reference SCG-D. SCG-D has status current from the effective time and predecessor reference SCG-C. Both records reference SUP-SERVICE-042.
  • Historical-use boundary: enquiries concerning service eligibility before the effective time go to a human reviewer with access to the historical repository. The current assistant must not silently apply revision D to those enquiries.

Affected passages and expected answer changes

  • Coverage area, section 2.1: revision C lists North and Central service zones. Revision D adds the East service zone. After cutover, a current enquiry about East-zone eligibility should use revision D, not the earlier exclusion.
  • Call-out charge, section 3.2: revision C describes one included call-out per service year. Revision D describes one included call-out per qualifying incident. Answers must preserve the qualifying-incident condition rather than shortening it to unlimited included visits.
  • Exclusions, section 4.3: revision D adds damage caused during third-party transport as an exclusion. The assistant should identify that exclusion without treating all transport-related questions as automatically excluded.
  • Contact route, section 6.1: unchanged. The extracted passage in revision C shares a chunk with the old call-out rule, so the complete old chunk must be retired; the current contact passage comes from revision D.

Transition timing and retirement work

  • By 11 September 2026 at 16:00 UTC: approve revision D, finish the passage map, and inspect SRC-118 plus the separately maintained call-out answer card.
  • Before the effective time: keep revision D in a preparation area excluded from live retrieval. Confirm that preview questions cannot expose its future rules through the current-answer route.
  • At 09:00 UTC on 14 September 2026: the authorized operator excludes SRC-118 and its derived passages, retires the old answer card, and activates SRC-164 for current answers.
  • During an unverified transition: route affected coverage questions to human review. Do not serve both revisions as equally current to avoid a temporary gap.
  • Completion condition: confirm active-source state and finish the retrieval and answer checks below before ending the temporary review route.

Post-replacement checks and hypothetical disposition

  • Check set: twelve current-rule queries, four historical-boundary queries, and four unchanged-rule queries, including paraphrases and misleading references to revision C.
  • Initial finding: one call-out paraphrase retrieves the retired answer card. The source replacement remains incomplete, even though the main guide is correct.
  • Corrective action: remove that card from active retrieval, confirm the relevant refresh has completed, and rerun the entire check set.
  • Illustrative final evidence: no predecessor passages appear in the inspected current-answer candidate results; current answers cite revision D; historical queries take the review route; unchanged contact answers remain correct.
  • Closure record: verification owner records query results, inspected source identifiers, observed completion time, and any visibility limits. These hypothetical findings are not customer results or a performance guarantee.

4. Separate approval, effectiveness, and retrieval readiness

Approval means the content has passed the organization’s review. Effectiveness means its rules now apply. Retrieval readiness means the intended source set has been checked and is available for the intended questions. These states can occur at different times.

A future-effective guide can be approved without being eligible for current answers. If the retrieval system cannot reliably enforce future dates, keep that guide outside live retrieval until cutover. A date printed in a header is not, by itself, a retrieval control.

At the other end, the business effective time may arrive before the technical change is verified. The new rule still applies; that does not make an unverified assistant answer safe to use. A temporary human-review route for affected questions is often clearer than allowing the assistant to choose between competing versions.

If the environment cannot make the replacement in one controlled step, explicitly choose how to handle the gap between exclusion and activation. Restrict the affected question class where possible rather than interrupting unrelated knowledge. Record who can authorize the transition and who can confirm completion.

5. Retire the predecessor at the retrieval layer

The retirement task is complete only when the old authority can no longer compete in the route used for current answers. Changing a title to archived or adding a replacement link may help a human reader, but it does not prove exclusion from retrieval.

  1. Change the source state. Retire or exclude the predecessor using the controls actually available in the environment.
  2. Address derived content. Check extracted passages, duplicate imports, answer cards, and other identified copies. Remove or rebuild affected material as necessary.
  3. Confirm propagation. Establish whether a refresh or rebuild is required, and inspect its completion rather than relying only on the original edit.
  4. Preserve the relationship. Keep the predecessor ID, replacement ID, effective boundary, and change record connected for later investigation.
  5. Inspect the active set. Confirm that the replacement is present and the predecessor is absent or ineligible for current-answer retrieval.

A replacement link is useful for traceability, but it is not a substitute for retirement. Likewise, an instruction such as prefer the latest version is weaker than preventing obsolete passages from entering the current-answer candidate set.

6. Test retrieval competition, not just one correct answer

A correct generated answer can conceal a failed retirement. The assistant might select revision D on one attempt while revision C remains among the available candidates. Test source selection as well as the final response.

Where diagnostic access permits, inspect retrieved passages and their source identifiers before answer generation. If the system exposes only final answers or citations, record that limitation: an answer-only check cannot establish that no obsolete candidate was retrieved.

  • Direct changed-rule questions: ask about each affected passage without naming a document.
  • Paraphrases: express the same eligibility issue in ordinary customer language.
  • Old-rule bait: include an outdated assumption and check whether the answer corrects it using current authority.
  • Date-boundary questions: distinguish current enquiries from historical eligibility questions.
  • Unchanged questions: confirm that retirement did not remove useful content that still belongs in scope.
  • Unsupported questions: check that the replacement is not stretched into answers it does not support.

Define acceptance criteria before running the checks. A practical criterion is no predecessor passages in the inspected current-answer results, correct use of the replacement’s conditions, and the agreed handling of historical questions. Passing a finite test set is evidence for that release decision, not proof that every possible query will behave correctly.

7. Recover without restoring a known obsolete rule

If an obsolete passage appears, keep the replacement record open. Capture the query, retrieved source ID, passage, and observed answer. Trace whether the source is the original guide, a duplicate, a mixed chunk, or a separate answer artifact. Fix the source path, then rerun the changed-rule and unchanged-rule checks.

If the replacement itself contains an error, distinguish a technical reversal from a business-rule reversal. After the effective time, restoring the predecessor may reintroduce rules that no longer apply. The safer response may be to suspend affected automated answers, obtain an approved correction, and publish a corrected replacement.

For an unexpected loss of unchanged information, restore that information through a current, approved source rather than reactivating the entire predecessor. This avoids repairing one missing contact instruction by bringing back several obsolete coverage rules.

Set a follow-up check after normal use resumes. Inspect examples that touch the changed passages and investigate contradictory answers as source-governance incidents, not merely wording problems.

8. Make the next replacement auditable

For the next revised source, create the supersession record before uploading it to live knowledge. Assign a change owner and verification owner, map every changed passage, and name the exact observable evidence that will close the change.

Your immediate deliverable is one completed record with no unexplained retirement locations: each identified copy is either excluded from current retrieval, replaced, or explicitly retained through a separate historical route. Pair that record with a saved check set and its results. That gives the next operator a repeatable decision trail—and a concrete way to detect when two active rules have survived the replacement.

AI knowledgeSource governanceDocument retirementRetrieval testing

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications