AI Search Over Controlled Documents: Making Sure Users Only See the Effective Version

Why Document Status Is the Real Risk in AI Search

Every regulated company already has a document control process. Procedures are written, reviewed, approved, given an effective date, trained on, periodically reviewed, revised, and eventually retired. That process exists so that a person on the floor or in the lab follows the current instruction and not an old one. Most of the work in a document management system (DMS) is about this one outcome: the right version, in the right hands, at the right time.

AI search puts a new layer between the user and that process. Instead of opening the DMS, finding the document, and checking the header, the user types a question into a chat window. The system retrieves passages from an index, hands them to a language model, and returns a written answer. If the index holds version 6 and version 7 of the same SOP, or holds a draft of version 8, the user may get an answer built from the wrong text and have no visible reason to doubt it.

What Document Control Already Requires

Nothing about this problem is new in regulatory terms. The rules that govern paper and electronic documents apply to any system that shows those documents to people. EU GMP Chapter 4 says the documentation requirements apply equally to all forms of document media types, and it states plainly: “The effective date should be defined.”¹ The same chapter says documents within the quality management system “should be regularly reviewed and kept up-to-date,” and that an inventory of documents within the quality management system should be maintained.¹

ICH Q7, the GMP guide for active pharmaceutical ingredients, is more direct about version handling: “The issuance, revision, superseding, and withdrawal of all documents should be controlled by maintaining revision histories.”² In its section on computerized systems, the same guide says those systems “should have sufficient controls to prevent unauthorized access or changes to data.”²

For nonclinical laboratories, 21 CFR 58.81 requires that each laboratory area have immediately available the laboratory manuals and SOPs for the procedures being performed, and that a historical file of SOPs and all revisions, including the dates of those revisions, be maintained.³ That last point matters later in this article. Regulators expect you to keep old versions, and they expect you to keep them from being used as current instructions. An AI search tool has to respect both expectations at once.

The Question Changes When Search Answers Instead of Lists

Traditional keyword search in a DMS returns a list of documents with their status shown next to each title. The user still does the last step: they see “Superseded” in the status column and skip that result. The control depends partly on the user, but the information needed to make the right choice is on the screen.

A generative answer removes that last step. The user sees a paragraph of instructions, often without the document header, the status, or the effective date. If a superseded passage made it into the retrieved context, it can be merged with current text into a single answer. The user cannot tell which sentence came from which version unless the system shows them. This is why the design has to move the status check out of the user’s head and into the retrieval pipeline.

Why the Model Will Not Save You

It is tempting to assume a capable model will notice that one passage is older and prefer the newer one. The research does not support that assumption. The ClashEval study built a dataset of more than 1,200 questions across six domains, including drug dosages, and changed the facts in the supporting documents. It found that six top-performing models adopted incorrect retrieved content, overriding their own correct prior knowledge, more than 60% of the time.⁴ The authors also found that models were less likely to adopt retrieved content the more unrealistic it was.⁴

That second finding is the uncomfortable one for document control. A superseded SOP is not unrealistic. It was correct once, it uses your company’s terms, and it may differ from the current version by a single hold time or one extra verification step. Based on that research, we would expect a model to accept it without hesitation.

Researchers call the situation where two retrieved passages disagree an “inter-context” knowledge conflict, one of three categories described in a 2024 survey of the topic.⁵ The survey makes clear that these conflicts affect the trustworthiness of model outputs and that there is no single solution. The original paper that introduced retrieval-augmented generation (RAG) in 2020 noted that providing provenance for model decisions and updating their knowledge were open research problems, which is part of why retrieval was attractive in the first place.⁶ Retrieval makes knowledge easier to update. It does not decide for you which knowledge is current.

The OWASP Top 10 for LLM applications makes a related point in its entry on vector and embedding weaknesses. It lists knowledge conflict errors that occur when data from multiple sources contradict each other, alongside unauthorized access and data leakage from misaligned access controls.⁷ Both risks come up in controlled document search, and both are addressed in the index, not in the prompt.

Mapping the Document Lifecycle Into the Index

Before anyone builds an index, quality and IT need to agree on how each lifecycle state in the DMS will be treated by AI search. Most DMS platforms have more states than people remember, and the names differ between systems and even between document types in the same system. The mapping has to be written down, approved, and kept under change control, because it is the specification the retrieval layer will be tested against.

A Starting Map of States

The table below is a starting point. Your states will differ, and the “default search” column is a decision for your quality unit, not a technical default.

Approved Is Not the Same as Effective

The most common mapping error we see is treating “approved” and “effective” as the same thing. Many companies approve a revised SOP days or weeks before it takes effect, so that training can be completed first. During that window, the old version is still the one people must follow. If the index picks up the new version on approval, AI search will start giving answers based on a procedure no one has been trained on and that is not yet in force.

The fix is simple to describe: the retrieval filter uses the effective date and time, not the approval event. It is harder to build than it sounds, because the effective date has to be stored as a proper date field in the index, the time zone has to match how the DMS defines “effective,” and the filter has to be evaluated at query time rather than baked in at indexing time. A document that is indexed on Monday with an effective date of Thursday must start appearing on Thursday without anyone re-running a job.

One Effective Version per Document Number

For most controlled document types, there should be exactly one effective version of each document number at any moment. That gives you a simple invariant to enforce and test: for every document number in the default index, count the distinct versions. The answer should always be one. If it is two, something in the sync has failed and you have a version conflict waiting to surface in an answer.

There are legitimate exceptions. Some companies keep site-specific variants under one parent number, or allow a local work instruction to coexist with a global SOP. Those cases need their own rule in the mapping, usually by making the site or scope part of the key. What matters is that the rule is explicit and testable, not left to whatever the indexer happens to do.

Metadata the Index Needs

At a minimum, every chunk in the index should carry the metadata needed to filter, cite, and audit it. Chunks are the passages a document is split into for retrieval, and each one needs to know where it came from. A practical minimum:

• Document number and version, exactly as shown in the DMS.
• Lifecycle state and the date and time it last changed.
• Effective date and, where used, the date the version was superseded.
• Document type (SOP, work instruction, specification, policy, form, and so on).
• Scope attributes such as site, function, product, or market, where documents are scoped.
• Access attributes such as the groups allowed to view the document.
• Section identifier so that a citation can point to a heading or step, not just to the whole file.
• A link to the record in the DMS, which is where the user should go to read the controlled copy.

This list looks obvious on paper. In practice, the quality of this metadata is where most of the effort goes. Legacy documents migrated years ago often have missing effective dates, inconsistent version labels, or document types that were never set. An AI search project is often the first time anyone has needed that metadata to be correct across the whole library, and a cleanup step is usually required before the index can be trusted.

Keeping the Index in Step With the Document Management System

An AI search index is a copy. It holds text and metadata extracted from the DMS, split into chunks, and stored with vector embeddings for semantic search. Like any copy of controlled content, it can drift from the source. EU GMP Chapter 4 asks that the reproduction of working documents from master documents not allow any error to be introduced through the reproduction process.¹ A search index is not a working document in the traditional sense, but the same concern applies: the copy must stay faithful to the master, including its status.

Pull the Status From the System of Record

The index should never decide status on its own. It should read status from the DMS and nothing else. That sounds obvious, but we have seen designs where status was inferred from a folder path, a file name suffix, or the “last modified” date on a file share. Each of these breaks the first time someone moves a file, renames it, or opens and resaves an old version.

Cloud platforms make it easy to fall into the “last modified” trap. Amazon Bedrock Knowledge Bases, for example, support metadata filters on document attributes, and AWS’s own documentation gives a modification timestamp field as an example that can be used to filter on the most recent data.⁹ That is a reasonable feature for general content. For controlled documents it is the wrong signal. A superseded SOP can be modified after the new version is effective (for example, when a watermark is applied or a record is migrated), and a current SOP can be years old. Filter on a status and effective date that come from the DMS, not on file timestamps.

Deletion Is Not Automatic

Many indexing tools are good at picking up new and changed content and poor at noticing that content should leave. Microsoft’s documentation for Azure AI Search is candid about this. It explains that change detection is built in for Azure Storage sources, but deletion detection is not, because an indexer does not track object deletion in the data source.⁸ Microsoft recommends a “soft delete” approach, where a flag marks the item for removal so the search document is deleted first and the physical file second. It also warns that “The deletion detection strategy must be applied from the very first indexer run,” because documents deleted before the policy is in place remain in the index even if the policy is added later.⁸

For controlled documents, “deleted” is usually the wrong word anyway. A superseded SOP is not deleted from the DMS; it changes state. So the design question is how a state change in the DMS becomes a removal (or a move to a historical index) in the search layer. Whatever tool you use, confirm in writing how it handles items that change state or disappear from the source, and test it before go-live.

The Sync Pattern We Recommend

There is no single correct architecture, but the pattern below has held up well in regulated settings because each step can be tested and each failure is visible.

Close the Gap Window

Even with a good sync, there is a window between a state change in the DMS and the matching change in the index. For most document types, a few minutes is acceptable. For a procedure that was superseded because the old version was unsafe, even that window may be too long. The query-time filter in step 3 closes most of this window, because a document with a future effective date appears on time and a document marked superseded disappears as soon as its status field changes.

Decide the acceptable window with your quality unit and write it into the requirements. Then measure it in testing by changing a document’s state in a test DMS and timing how long it takes for the change to show up in AI search results. That number belongs in your validation evidence.

Superseded, Draft, and Historical Documents: When Users Need Them

Excluding everything except effective documents is the right default, but it is not the whole answer. There are real, legitimate reasons to search old and unreleased documents. The goal is to support those uses on purpose, in a separate mode, with clear labels, rather than letting them leak into everyday answers.

Investigations Need the Version That Was in Effect

A deviation investigation often asks what the procedure said on the day the event happened. A batch record review may need the version of the master instruction the batch was made under. An auditor may ask what changed between two versions of a specification. In each case, the user needs the superseded version, and they need to know it is superseded. The GLP requirement to keep a historical file of SOPs and their revision dates exists precisely because this kind of question comes up.³

AI search can help here, and it can be very useful. “What did the cleaning SOP for line 3 require for rinse sampling in March 2025?” is a good question for a retrieval system, provided it filters on the version effective on that date rather than on the current version. That is a different filter from the default one, and it should be a deliberate choice by the user, not something the system switches to on its own.

Drafts Belong to Authors and Reviewers

Authors revising an SOP may want AI help to compare their draft against the current version, check consistency with related procedures, or find every other document that mentions a changed term. That is a reasonable use. It should happen in a space limited to the people working on the draft, with an index that holds the draft and the related effective documents. It should never put draft text into answers for general users, because draft content may change before approval or never be approved at all.

Four Modes, Kept Separate

We recommend designing AI search around a small number of named modes, each with its own filter, labels, and access rules. The user always knows which mode they are in.

What Happens When There Is No Effective Source

In current mode, a user will sometimes ask about a procedure that has been retired, or about a topic that no effective document covers. The worst outcome is an answer built from general model knowledge that sounds like company policy. The second worst is an answer that reaches into a superseded document because nothing else matched.

The better behavior is to say plainly that no effective controlled document covers the question, and where appropriate, that a related document was retired on a given date. That requires the system to know about retired documents (so it can say so) without using their content in the answer. It is a small design detail that makes a large difference to user trust, and it should be written into the requirements and tested.

Access Control: Permission to Open Is Not Permission to Retrieve

Access control for AI search has two parts that are easy to confuse. The first is whether a user is allowed to see a document at all. The second is whether the document, even if they can see it, should be used to answer their question. Most off-the-shelf tools handle the first reasonably well. The second is the one that matters most for controlled documents, and it usually needs to be designed.

View Rights Are Broader Than You Think

Microsoft’s documentation for its Copilot products states: “Microsoft Copilot only surfaces organizational data to which individual users have at least view permissions.”¹⁰ Microsoft adds that its semantic index honors the user’s identity-based access boundary, so grounding only uses content the current user is authorized to access.¹⁰ That is the right behavior for security, and it is also a warning for document control.

In many companies, people have view access to far more than the effective library. Collaboration sites hold working copies of SOPs downloaded for a training session. Project folders hold drafts from a revision that finished two years ago. Shared drives hold PDF exports of documents that have since been superseded. If an AI assistant grounds on everything a user can view, it will find these copies, and none of them carry the status that the DMS carries. A permissions model built for sharing files was never designed to separate current instructions from stale copies.

The practical conclusion is that general workplace assistants should not be the channel for answers about controlled procedures unless the content they can reach has been cleaned up and scoped. A purpose-built assistant that grounds only on an index fed from the DMS is easier to control, easier to validate, and easier to explain to an inspector.

Filters Are Not Authorization Unless You Make Them So

Many retrieval platforms implement document-level access by storing the allowed user or group identifiers on each indexed item and filtering results at query time. Microsoft describes this “security filter” pattern for Azure AI Search and is explicit that the principal stored in the index is only a string used in a filter expression. The filter does not authenticate or authorize anyone by itself.¹¹ The application must get the caller’s identity from a trusted source and apply the filter on the server, every time.

The same documentation set describes newer built-in options, including ingesting SharePoint permissions and Microsoft Purview sensitivity labels and evaluating them at query time. At the time of writing, Microsoft lists several of these capabilities as preview features, which it says are not recommended for production workloads.¹² Check the current status of any such feature before you rely on it in a GxP system, and record that check in your supplier assessment.

Status Is an Access Attribute Too

A helpful way to design this is attribute-based access control (ABAC). NIST defines ABAC as an approach where authorization is decided by evaluating attributes of the user, the object, the requested operation, and sometimes the environment, against a policy.¹³ For AI search over controlled documents, the relevant attributes usually include:

• User attributes: role, site, function, training status, and whether the user is an author or reviewer on a given document.
• Document attributes: lifecycle state, effective date, site or market scope, confidentiality class, and document type.
• Operation: current mode, point-in-time mode, authoring mode, or comparison mode.
• Environment: the current date and time, which drives the effective date filter.

Seen this way, “only show effective versions” is not a separate feature bolted onto access control. It is one rule in the same policy: in current mode, a user may retrieve a chunk only if they have view rights to the document and the document is effective now. Writing it as one policy makes it easier to specify and to test.

Where Leaks Happen Outside the Index

OWASP recommends permission-aware vector stores and strict logical partitioning of data so that one group’s content is not returned for another group’s queries.⁷ Those controls cover the index. Leaks also happen in places teams forget:

• Answer caches. If answers are cached by question text, a cached answer built for one user can be served to another user with different rights, or served after the source document was superseded.
• Conversation history. A user in authoring mode asks about a draft, then switches to current mode in the same conversation. The draft text is still in the conversation history the model sees.
• Logs and analytics. Prompt and response logs can hold restricted content and need the same access rules as the documents.
• Suggested questions and summaries. Features that pre-generate summaries or “popular questions” from the library can expose restricted content to everyone.

Each of these needs a rule. Caches should key on user permissions and document versions, and be cleared when a version changes. Mode switches should start a new context. Logs should be access-controlled and covered by your retention procedure.

Citations Back to the Controlled Source

A citation is how a user checks an answer and how a reviewer reconstructs what happened later. In controlled document search, it also does a third job: it keeps the DMS as the source of truth. The answer is a pointer. The controlled document is what the user follows.

Citations Are Often Wrong, Even When Present

Showing citations is not the same as showing accurate citations. A study of four popular generative search engines found that on average only 51.5% of generated sentences were fully supported by their citations, and only 74.5% of citations supported the sentence they were attached to.¹⁴ The authors noted that the responses were fluent and appeared informative, which is exactly why the gaps were hard for users to notice.¹⁴

A later, preregistered study of AI legal research tools built on RAG found that the RAG-based products tested each hallucinated between 17% and 33% of the time, even though vendors had marketed retrieval as removing or avoiding hallucinations.¹⁵ Legal research is not GMP, but the setting is similar: a closed body of authoritative text, users who rely on precise wording, and serious consequences for an error. The lesson carries over. Retrieval reduces errors; it does not remove them, and citations have to be tested, not assumed.

What a Good Citation Looks Like

For controlled documents, a citation should let a user go from the answer to the exact controlled text in one click, and should tell them the status of that text without opening anything. A good citation includes:

• Document number and title.
• Version and effective date.
• Section or step number the answer relies on.
• A link to the document record in the DMS, not to a copy held by the search tool.
• A status label if the answer comes from anything other than current mode.

Linking to the DMS record matters more than it seems. If the citation opens a copy of the document stored by the search platform, you now have an uncontrolled copy that users can print, save, and forward. If it opens the record in the DMS, the user sees the controlled version with its official header and watermark, and the DMS’s own controls apply.

Keep Answers Short and Close to the Source

Long answers that combine many passages are harder to cite accurately. Research on how models use long inputs found that performance is often highest when relevant information appears at the beginning or end of the input, and degrades significantly when the model must use information in the middle of a long context.¹⁶ For controlled documents, that argues for retrieving fewer, more precise passages, keeping answers short, and quoting the procedure’s own wording for critical steps instead of paraphrasing it.

We also recommend a plain statement near every answer that the controlled document governs. It is not legal cover. It is a reminder of how the system is meant to be used: to find and understand the right procedure faster, not to replace reading it.

Validating Retrieval, Not Just Answers

Most AI validation discussions focus on the quality of the generated answer. For AI search over controlled documents, the retrieval layer is where the GxP-relevant behavior lives, and it is also the part that can be tested most precisely. Whether an answer is well written is partly a matter of opinion. Whether a superseded chunk was returned is a yes or no question.

Start With Intended Use and Risk

The first validation decision is how the tool will be used and what happens if it is wrong. An assistant that helps new staff find the right SOP and then sends them to read it carries a different risk from one that answers operators at the point of use during manufacturing. The draft EU GMP Annex 22 on artificial intelligence, published for consultation in 2025 and not yet final, states that it does not apply to generative AI and large language models, and that such models should not be used in critical GMP applications. For non-critical GMP uses, the draft says qualified and trained personnel should always be responsible for making sure the outputs are suitable for the intended use.¹⁷ EMA has since returned to the generative AI question in expert discussions, so this is an area to watch rather than a settled rule.

Whatever the final text says, the direction is clear enough to design against. Keep AI search in a supporting role, where a trained person reads the controlled document before acting, and document that intended use. That puts the tool in a lower risk category and makes the validation scope manageable.

Tests That Target Document Status

General RAG evaluation frameworks are useful, but they do not know anything about your document lifecycle. RAGAS, for example, offers reference-free metrics for how well retrieval finds relevant, focused passages and how faithfully the model uses them.¹⁸ Those metrics tell you whether an answer is supported by what was retrieved. They cannot tell you whether what was retrieved was the effective version. You need tests built for that.

Set Acceptance Criteria That Match the Risk

Some of these tests have a natural acceptance criterion of zero failures. A version trap test that returns superseded text in current mode is a failure of the core control, not a statistical variation, and one occurrence should stop release until the cause is found. The same applies to draft isolation and access boundary tests.

Other tests, such as citation accuracy and answer quality, are measured on a sample and need a threshold agreed in advance with the quality unit. Be careful not to borrow a threshold from a vendor benchmark. Set it based on your intended use, and document why that number is acceptable for a supporting tool where a trained person reads the procedure before acting.

Keep Testing After Go-Live

The retrieval layer changes more often than most validated systems. The DMS is updated, new document types are added, the embedding model is replaced, the chunking settings are tuned, the language model version changes. Each of these can change which passages are retrieved. Keep the status-focused test set as a regression suite, run it on every change to the pipeline, and run it on a schedule even when nothing has changed, because the document library itself changes every week.

The daily reconciliation job described earlier is part of this ongoing evidence. So is monitoring of user feedback: a “this answer is wrong or out of date” button, routed to a named owner, turns users into an early warning system for sync failures. Review those reports in the same forum that reviews other quality signals.

A Practical Rollout Sequence

The order in which work is done matters as much as the design. Teams that start with the chat interface and add status controls later usually find themselves rebuilding the index. The sequence below puts the controlled document work first, because everything else depends on it.

Who Owns What

AI search over controlled documents spans several owners, and unclear ownership is a common reason these projects stall. A workable split: the quality unit owns the state map, the intended use, the acceptance criteria, and the review of reconciliation findings. The DMS owner owns metadata quality and the event feed. IT or the platform team owns the index, the sync, the filter, and the access integration. The business process owners own the question set used in testing, because they know which procedures change often and which details matter most.

One more point on ownership. When a document owner revises a procedure, they should know that AI search will pick up the change and that the old version will stop being used in answers on the effective date. Adding a line to the document change procedure that confirms this (and names who checks it) connects the AI tool to the document control process it depends on, instead of leaving it as a separate system that quality has to remember.

What to Ask a Vendor

If you are buying rather than building, the same questions apply. Ask the vendor to show, in a test environment with your DMS, how the product handles:

• A document approved today with an effective date next week.
• A version superseded while a user has a conversation open.
• A retired procedure with no replacement.
• A draft that shares a document number with an effective version.
• A user whose site access changes.
• The index falling out of step with the DMS, and how you would find out.

Ask where the status filter is applied, whether any of the access features they rely on are in preview, and what evidence they can give you for your validation package. The answers will tell you quickly whether the product was designed for controlled content or adapted from general enterprise search.

Conclusion

AI search over controlled documents is worth doing. It can shorten the time it takes a new analyst to find the right method, help an investigator compare what a procedure said before and after a change, and make a large procedure library easier to use. But the value depends entirely on one property that general-purpose tools do not provide by default: the answer comes from the effective version, for a user allowed to see it, with a citation that leads back to the controlled record. The model cannot supply that property. The index, the sync, the filters, and the tests have to.

The good news is that none of this requires new regulation or new theory. It requires taking the document control process you already run, writing it down as retrieval requirements, and testing those requirements as precisely as you would test any other GxP control. Sakara Digital works with pharma and biotech organizations on exactly this kind of AI and quality design. If you are planning AI search or chat over your controlled documents and want an independent view on where the status and access controls should live, we are happy to have that conversation.

For Further Reading