Validating OCR and Document Extraction AI for GxP Records

Why Document Extraction Needs Its Own Validation Approach

Most AI validation guidance is written with a classifier or a prediction model in mind: the model looks at data and returns a decision such as accept or reject. A document extraction tool does something different. It reads a picture of a page and writes values into a system of record. Its output is not a decision about the data. Its output is the data. That changes what can go wrong and what you need to prove.

A Hybrid System by Definition

Any process that starts with paper or a scanned image and ends with structured data in a computerized system is a hybrid system. The WHO guideline on data integrity uses that term for any combination of paper and electronic formats, and it asks companies to design processes that avoid unnecessary transcription of data and unnecessary conversion from paper to electronic and back.¹ It also says the use of legacy hybrid systems should be discouraged, with a priority timeline set for replacing them.¹ An extraction tool does not remove the hybrid step. It automates the conversion, which makes the conversion itself something to validate.

The draft revision of EU GMP Annex 11 takes a similar line. Where critical data must move from one system to another, it should move through validated interfaces where possible rather than through manual transcription, and if critical data is transcribed by hand, effective measures should be in place so the transcription does not create a data integrity risk.² An AI extraction tool fits neither category neatly. It is not a validated interface between two systems, and it is not a person typing. It is automated transcription with an error profile of its own, and that profile has to be measured rather than assumed.

The Errors Are Different From Human Errors

People who transcribe records make errors that are mostly random: a transposed digit late in a shift, a skipped line. Second-person verification was designed for that kind of error. AI extraction errors behave differently. They tend to be systematic, tied to a layout, a font, a writer’s handwriting, or a scanner. If a tool misreads one supplier’s certificate layout, it may misread every certificate from that supplier in the same way. And the tool usually reports its wrong answer in the same clean format as its right ones.

Why a Vendor Accuracy Figure Is Not Evidence

Vendors often quote a single accuracy number. That number is usually measured on the vendor’s own benchmark documents, often at the character level, and averaged across every field. Your risk is concentrated in a small number of critical fields on your hardest documents. A tool can read 99 percent of characters correctly and still get a meaningful share of handwritten lot numbers wrong, because one wrong character makes the whole lot number wrong.

FDA’s data integrity guidance makes the general point plainly. A CGMP workflow is an intended use of a computer system to be checked through validation, and “If you validate the computer system but you do not validate it for its intended use, you cannot know if your workflow runs correctly.”³ Draft Annex 22 adds that documentation of model training, validation, and testing should be available to and reviewed by the regulated user whether the work was done in-house or by a supplier.⁴ A vendor’s figure can inform your plan. It cannot replace your own test on your own documents.

Start With Intended Use and a Field Inventory

Every later choice in the validation depends on a clear statement of what the tool is for. For document extraction, that statement has to be more specific than “extract data from batch records.” It has to say which documents, which fields, which downstream decisions, and what happens to the original.

What the Intended Use Statement Must Say

Draft Annex 22 asks for the intended use of a model to be described in detail, based on in-depth knowledge of the process it supports. That description should characterize the input data the model will see, including all common and rare variations (the draft calls this the input sample space), and it should identify limitations and inputs that could be erroneous or biased. A process subject matter expert should be responsible for the description, and it should be documented and approved before acceptance testing starts.⁴

For an extraction tool, a usable intended use statement covers at least the following:

• Document types and versions. Executed batch record pages for named products and master record versions, supplier certificates of analysis for named materials, equipment use logbooks, cleaning records.
• Sources. Which sites, which contract manufacturers, which suppliers. Each supplier’s certificate layout is, in practice, its own document type.
• Capture methods. Flatbed scanner, multifunction printer, phone photo, fax, native PDF from a supplier portal.
• Fields and their use. Every field the tool extracts, and what each value feeds: a release decision, a trend chart, a search index.
• Human role. Who reviews what, and what they are responsible for.
• Exclusions. Document types or conditions the tool is not approved to process, and what happens when one arrives.

FDA’s draft guidance on AI for regulatory decision-making offers a helpful way to think about how much rigor each use needs. It describes model risk as a combination of model influence (how much the AI output contributes relative to other evidence) and decision consequence (how serious an adverse outcome from a wrong decision would be).⁵ That guidance is still a draft, and it excludes AI used for operational efficiencies that do not affect patient safety, drug quality, or the reliability of study results.⁵ An extraction tool that writes certificate results used for material release does affect quality, so the two-factor view is a reasonable lens even before the guidance is final.

Tier the Fields, Not the Documents

A single batch record page holds fields of very different importance. Treating the page as one unit with one accuracy target either overspends effort on low-risk fields or underprotects the critical ones. A field inventory with risk tiers solves this.

The Tier 1 list for batch records can start from the regulation. In the United States, 21 CFR 211.188 requires batch production and control records to document, among other things, the weights and measures of components used, in-process and laboratory control results, and a statement of the actual yield.⁶ For certificates of analysis, 21 CFR 211.84(d)(2) allows a manufacturer to accept a supplier’s report of analysis in place of its own testing, provided it runs at least one specific identity test and establishes the reliability of the supplier’s analyses through appropriate validation of the supplier’s results at appropriate intervals.⁷ If an extraction tool feeds those supplier results into the comparison, an extraction error weakens the evidence the reduced testing program depends on.

Do not dismiss Tier 3. A batch number misread at indexing can file a record where nobody will look for it. The data is still there, but it is no longer available in any practical sense.

Measure the Process You Are Replacing

Draft Annex 22 includes a “no decrease” principle: the acceptance criteria for a model should be at least as high as the performance of the process it replaces, which means the performance of the current process has to be known.⁴ At many sites, nobody has measured the manual transcription error rate. What is known is that second-person verification exists.

Establishing the baseline is usually simple and worth doing before anything else. Take a sample of recent transcriptions that were second-person verified, count how often the verifier made a correction, and then re-check a smaller sample of the verified values against the source to estimate what verification missed. The result gives you a defensible number for Tier 1 acceptance criteria. Without it, you are either guessing at a target or adopting a vendor’s.

Know Which Kind of Model You Are Validating

Two tools can look identical on screen and be built very differently underneath. The difference matters a great deal under the current EU draft, and it changes how the tool fails.

Two Architectures Behind the Same Screen

What Draft Annex 22 Says About Each

Draft Annex 22 applies to computerized systems used in the manufacture of medicinal products and active substances where AI models are used in critical applications, meaning those with a direct impact on patient safety, product quality, or data integrity.⁴ It covers static models, which do not adapt during use, and models with deterministic output, which give identical outputs for identical inputs. It states that models with probabilistic output are not covered and should not be used in critical GMP applications, and that the document does not apply to generative AI and large language models, which also should not be used in critical GMP applications. If such models are used in non-critical GMP applications, qualified personnel should always be responsible for confirming the outputs are suitable, which the draft calls human-in-the-loop.⁴

An extraction tool that writes executed batch record values or certificate results into a GMP system has a direct effect on data integrity. In most cases, that makes it a critical application under the draft’s own definition. A tool used only to suggest search tags for an archive, with the original documents still the basis for every decision, is a different case and may be non-critical.

The scope is worth noting for another reason. Draft Annex 22 is a manufacturing annex. Extraction tools used on clinical or pharmacovigilance documents are outside its direct scope, although the same testing principles transfer well.

Status: Still a Draft, and the Generative AI Line Is Under Review

The European Commission’s consultation on draft Annex 22, together with revised Chapter 4 and Annex 11, opened on 7 July 2025 and closed on 7 October 2025.⁸ As of 29 October 2026, the Commission’s EudraLex Volume 4 page lists no Annex 22 and still shows Annex 11 in its January 2011 revision.⁹ There is no final text and no date for it to come into operation. The binding EU text for computerized systems today is the 2011 Annex 11.

The position on generative models may also change. EMA held a two-day workshop on 30 June and 1 July 2026 because the 2025 consultation suggested support for enabling generative AI and probabilistic models in manufacturing, and the Annex 22 drafting group sought expert input on control and mitigation measures such as guardrails.¹⁰ In the United States, there is no AI-specific GMP rule; 21 CFR Parts 11 and 211 apply to the records the tool creates. In short, draft Annex 22 describes the direction regulators are taking, not a requirement in force. It is still the most detailed public statement of what an inspector will expect to see for an AI model in a GMP process, which is why this article uses it as the reference point.

Why the Distinction Matters for Extraction

Research on multimodal language models shows the specific risk. A 2025 study introduced a benchmark for what its authors call OCR hallucination in degraded documents, built from identity cards and invoices with simulated real-world degradation. The authors report that current models often fail to recognize visual degradation and ambiguity, rely too much on language patterns, and generate hallucinated content when a precise answer is not possible.¹¹ A classic OCR engine that cannot read a smudged digit usually produces a visible error or a low confidence score. A language model may produce the digit it expects to see.

That leaves three workable positions for critical fields today:

• Choose a deterministic architecture for critical fields and validate it against draft Annex 22 as written.
• Use a generative tool as a suggestion only, with a person verifying every critical field against the source. The draft notes that where a model gives input to a human decision and testing effort has been reduced on that basis, review of every output may be needed, with records kept.⁴ In this design the person, not the model, is the control, and the efficiency gain comes from faster review rather than fewer reviews.
• Limit a generative tool to non-critical uses, such as indexing or search, until the final text settles the question.

The ISPE GAMP Guide: Artificial Intelligence, published in July 2025 for use alongside GAMP 5 (Second Edition), gives lifecycle guidance that applies to any of these designs.¹² Whatever the architecture, include a determinism test in the validation: run the same test documents through the tool twice and confirm the outputs match. If they do not, you have learned something important about the tool before an inspector does.

Building an Independent Test Set From Real Document Variety

The test set is where most extraction validations succeed or fail. A test set of clean, recent, typed documents will produce excellent numbers and prove very little. The documents that cause errors in production are the old scans, the faxed certificates, the pages with three corrections and a stamp across the result column.

Map the Sample Space Before You Sample

Draft Annex 22 asks for the input sample space to be divided into subgroups where applicable, based on relevant characteristics such as site, equipment, material, or the type of task.⁴ Test data should represent the full sample space of the intended use, be stratified, include all subgroups, and reflect common and rare variations, with the selection criteria and rationale documented.⁴ For documents, the subgroups are mostly visible on the page.

Some of these dimensions will need to be acceptance subgroups with their own criteria (handwritten Tier 1 fields, for example). Others are coverage dimensions, where the goal is to confirm the tool does not fail badly on an unusual condition.

Use Real Documents, Including the Bad Ones

Pull hard cases from the archive deliberately: pages from the oldest scanner, records known for difficult handwriting, faxed certificates, documents from the supplier everyone complains about. Draft Annex 22 says generation of test data or labels, for example with generative AI, is not recommended and any use should be fully justified.⁴ Synthetic degradation of clean images can be useful as a stress test during development. It should not be the evidence you rely on for acceptance.

The draft also says that any cleaning or exclusion of test data should be documented and fully justified, and that any pre-processing should be pre-specified with a rationale that it represents intended use conditions.⁴ For extraction, this means you cannot drop the pages the tool struggled with, and you cannot deskew or enhance test images more than production images will be.

Ground Truth Needs Its Own Control

Draft Annex 22 expects test labels to be verified through a process that ensures a very high degree of correctness, which may include independent verification by multiple experts.⁴ For extraction, the label is the correct value of each field. A practical approach is double independent keying: two trained people each record the value of every test field from the original paper where it still exists, the system compares them, and a third person settles any disagreement against the original.

Two details are often missed. First, “unreadable,” “blank,” and “N/A” are valid expected values. If a field cannot be read on the original, the correct tool behavior is to flag it, and the label should say so. Second, for corrected entries, the label is the final corrected value, and the test should also check that the tool flags the presence of a correction, because the reviewer needs to know one was made.

Keeping the Test Set Independent

Draft Annex 22 Section 6 is one of its most specific parts. It asks for technical or procedural controls that make sure data used to test a model was not used during its development, training, or validation. It allows two approaches: capturing test data only after training and validation are finished, or splitting test data from the full data pool before training begins. If test data is split off, people involved in developing and training the model should never have had access to it, the test data should be protected by access control and an audit trail, and there should be no copies outside the repository. The draft also asks that it be recorded which data was used for testing, when, and how many times.⁴

Two further clauses translate directly to documents. Where test data comes from physical objects, the objects used for the final test should not have been used to train or validate the model unless the features are independent. And staff who have had access to test data should not be involved in training and validating the same model; where that is impossible in an organization, the draft says they should only work on training and validation in pairs with a colleague who has not had access.⁴

Where Leakage Happens With Documents

With extraction tools, test data contamination rarely looks like a data scientist copying files. It looks like ordinary configuration work:

• Template and prompt tuning. The person who drew extraction zones on a batch record template, wrote field rules, or tuned prompts for a generative tool looked closely at specific pages. Those pages are development data. They cannot also be test data.
• Splitting at the wrong level. Fields from the same page share a writer, a pen, and a scan. Pages from the same batch record share even more. If you split the test set at the field or page level, the tool is tested on near-copies of what it was tuned on. Split at the document level at a minimum. For certificates, keep a separate set of entire suppliers held out, so you can measure how the tool handles a layout it has never seen.
• Re-scanning. A new scan of a page used during configuration is a new image of the same physical object. It is not independent test data, in the same sense the draft applies to physical objects.
• Threshold tuning on the test set. If confidence thresholds are adjusted after looking at test results, the test set has become development data. Use a separate calibration set for thresholds.
• Vendor training. Ask whether customer documents are used to train or improve shared models. If your documents could reach the vendor’s training data, you need to know before you assemble a test set from them.

Practical Controls

None of this requires special technology. A locked repository with an access log, a named custodian who is not on the configuration team, a manifest of file hashes for every test document, and a usage log recording each test run cover most of what the draft describes. Plan for one final acceptance run on the independent set. If the run fails and the tool is changed, document the deviation and plan how you will test again, ideally with newly collected documents.

Field-Level Metrics and Acceptance Criteria

The right metric for an extraction tool is not the one vendors usually report. Character error rate and average field accuracy describe how the tool performs in general. Your acceptance decision depends on how often a wrong value would reach the system of record without anyone checking it.

Count Outcomes, Not Characters

The unit of measurement should be the field instance: one field on one document. A lot number with one wrong character is a wrong field, whatever the character error rate says. Each field instance then falls into one of four outcomes once routing is applied.

Draft Annex 22 asks for suitable, case-dependent test metrics and gives a confusion matrix, sensitivity, specificity, accuracy, precision, and F1 score as examples for a classification model.⁴ For an extraction tool, this four-outcome table is the confusion matrix that matters. From it you can calculate:

• Undetected error rate: wrong and auto-accepted, divided by all auto-accepted field instances. This is the main acceptance metric for Tier 1.
• Automation rate: the share of field instances auto-accepted. This tells you how much review effort the tool removes.
• Catch rate: the share of wrong values that were routed to review. A low catch rate means the thresholds and checks are not finding the errors.
• Raw field accuracy: the share of field instances the tool got right before routing. Useful for comparing tools, less useful for acceptance.

“Correct” needs a written definition before testing starts. Decide in advance how values are normalized for comparison: whether “1,5” and “1.5” match for a European certificate, how dates are compared, whether trailing zeros matter. Draft Annex 22 expects pre-processing to be pre-specified with a rationale, and the same discipline applies to comparison rules.⁴ A comparison rule written after seeing results is a way of changing the answer.

Set Criteria by Tier and Subgroup

The draft expects acceptance criteria to be set by a process subject matter expert, documented, and approved before acceptance testing, and it allows criteria to differ for specific subgroups.⁴ For extraction, that usually produces a small matrix. The figures below illustrate the shape of the criteria. They are not regulatory numbers, and your own will depend on the baseline you measured and the risk of each field.

• Tier 1, each subgroup: the upper 95 percent confidence bound of the undetected error rate is at or below the measured baseline for second-person-verified manual transcription. Every fabricated value (a value returned for a field labeled blank or unreadable) is investigated as a failure, whatever the overall rate.
• Tier 2: undetected error rate below a set limit, with catch rate above a set limit, assessed across the whole set and for handwritten fields separately.
• Tier 3: document type and batch index correct for retrieval, and page counts reconciled for every document.
• All tiers: identical outputs on a repeat run of the same documents.

How Many Fields You Need

Sample size is where many extraction validations come up short. Draft Annex 22 says the test set and each of its subgroups should be large enough to calculate the test metrics with adequate statistical confidence.⁴ For an error rate, that means a confidence interval for a proportion. The NIST/SEMATECH e-Handbook of Statistical Methods describes the standard approach and its formulas.¹³

The table below uses a standard exact binomial calculation to show how many field instances you need in a subgroup to claim, with 95 percent confidence, that the true undetected error rate is below a target.

Two practical points follow. First, the counts apply per subgroup when the criterion is per subgroup. If handwritten Tier 1 fields from each of four sites have their own criterion, each needs its own sample. Second, small test sets cannot prove very low error rates. If the honest sample available supports only a 1 percent bound, the design should send more Tier 1 fields to human review rather than claim a precision the evidence does not support. That is a legitimate validation outcome, not a failure.

Note that the undetected error rate is calculated on auto-accepted fields only. If the tool auto-accepts 40 percent of Tier 1 fields, you need enough auto-accepted instances to meet the count, which means a larger test set overall.

Failures and Deviations

The draft expects a test plan prepared and approved before testing, containing the intended use, the metrics and acceptance criteria, a reference to the test data, a test script, and a description of how metrics are calculated. Any deviation from the plan, failure to meet criteria, or omission of test data should be documented, investigated, and fully justified.⁴ The temptation with extraction tools is to fix a failing field rule and rerun the same test set. Doing that without a documented deviation turns the test set into development data and leaves you with results you cannot defend.

Confidence Thresholds and Human Review Routing

Routing is what turns an extraction model into a controlled process. The model reads the page; the routing design decides which values a person checks. In a well-designed tool, most of the risk control comes from this step.

Confidence Scores Are Not Probabilities Until You Check

Most extraction tools report a confidence score for each field. It is tempting to read a score of 0.95 as “95 percent likely to be right.” That reading is often wrong. A widely cited study found that modern neural networks are poorly calibrated, meaning their confidence scores do not match how often they are correct, and that a simple adjustment called temperature scaling corrects this well on most datasets.¹⁴ You cannot assume the vendor has calibrated the scores for your documents.

The fix is empirical. On a calibration set (separate from the final test set), group field instances by confidence band and measure the actual accuracy in each band, separately for each tier and for handwritten versus printed fields. You will often find that the scores mean different things for different field types. Draft Annex 22 asks that during testing the system log the confidence score for each prediction where applicable, and that models have a threshold so outputs are produced only when suitable, and that, when the confidence score is very low, it be considered whether the model should flag the result as undecided rather than give a potentially unreliable answer.⁴

Choosing the Threshold

Research on selective classification, sometimes called the reject option, formalizes the trade. A model can be allowed to decline cases so that the error rate on the cases it does answer stays at a level the user sets; the trade-off is lower coverage.¹⁵ For extraction, coverage is the automation rate and declined cases go to a person.

Set thresholds per tier on the calibration set, choosing the lowest threshold at which the error rate among accepted fields meets the target. Then confirm the result on the independent test set without further adjustment. Expect Tier 1 thresholds to send most fields to review at first. That is acceptable. Production monitoring will produce the evidence to lower the threshold later, under change control.

Confidence Is Not the Only Signal

A confidence score measures how sure the model is. It does not measure whether the value makes sense. Rule checks that run on every extracted value catch a different set of errors:

• Format checks: a lot number matches the expected pattern; a date is a real date.
• Plausibility checks: a weight or result is within an expected range. The draft Annex 11 revision expects plausibility checks on manually entered critical data where applicable, and the same idea applies to machine-entered data.²
• Master data checks: the material code and lot number exist in the ERP or LIMS record for that receipt.
• Cross-field checks: the result and the specification are consistent with the pass or fail statement on the certificate; component weights sum as expected.
• Completeness checks: the page count matches the document’s own page numbering; every expected field has a value or a flag.

Any failed check sends the field to review regardless of its confidence score. Be clear about the limits, though. A plausibility check cannot catch a plausible wrong value, and a high-confidence wrong value that passes every rule is exactly the undetected error your acceptance criteria measure. Confidence, rules, and sampled verification work together. None is enough alone.

Designing Review So People Catch Errors

Routing a field to a person only helps if the person checks it. A systematic review of automation bias, the tendency to over-rely on automated output, drew on research from several fields with a focus on healthcare decision support. The review identified trust, workload, task complexity, and time pressure among the factors that influence it, and listed training, emphasizing user accountability, the position of advice on the screen, and updated confidence levels attached to system output among the factors that reduce it.¹⁶ The study was not about document extraction, but a reviewer confirming pre-filled values under time pressure is a typical setting for it.

Review screens should show the cropped region of the source image beside each value, with the field highlighted on the full page one click away. For the highest-risk Tier 1 fields, consider blind entry: the reviewer keys the value from the image without seeing the tool’s answer, and the system compares the two. Avoid a single “confirm all” button for a page. Track each reviewer’s override rate, because a reviewer who never corrects anything is either reviewing perfect output or not reviewing.

Draft Annex 22 makes the human role part of the validated design. Where a model gives input to a human decision and testing effort has been reduced on that basis, the intended use should describe the operator’s responsibility, and the operator’s training and consistent performance should be monitored like any other manual process.⁴ Records of that review should be kept.⁴

The Audit Trail and ALCOA+: Keeping Every Value Tied to Its Source

An extracted value without a link to its source is an unattributed number. The link from each value back to a specific region of a specific page is what makes extracted data defensible, and it is the first thing an investigator will ask to see.

The Source Image Is the Record That Matters

MHRA’s GxP data integrity guidance defines ALCOA as attributable, legible, contemporaneous, original, and accurate, with the “+” adding complete, consistent, enduring, and available.¹⁷ FDA’s data integrity guidance uses the ALCOA terms and says complete, consistent, and accurate data should be attributable, legible, contemporaneously recorded, original or a true copy, and accurate.³

The MHRA guidance also defines a true copy as a copy verified, by a dated signature or by generation through a validated process, to have the same information as the original, including the data that describes its context, content, and structure. It allows true copies to be kept in place of the original, a scan of a paper record being its example, if a documented system is in place to verify and record the integrity of the copy.¹⁷ FDA similarly accepts electronic copies as true copies of paper or electronic records if they preserve the content and meaning of the original, including all metadata needed to reconstruct the CGMP activity.³

Two consequences follow for extraction. First, scanning and extraction are separate steps with separate controls. The scan has to be a controlled true copy process on its own terms: complete pages, adequate resolution, verified before the paper is destroyed if the paper is to be destroyed at all. Second, the extracted value is derived data. It does not replace the source image as the original. If the extracted value and the image ever disagree, the image is authoritative, and the system must make the image easy to reach.

What the Audit Trail Must Show

Under 21 CFR 11.10(e), closed systems must use secure, computer-generated, time-stamped audit trails to independently record the date and time of operator entries and actions that create, modify, or delete electronic records, and record changes must not obscure previously recorded information.¹⁸ The draft revision of Annex 11 spells out who, what, when, and why: the user who made a change (with their role where users can hold more than one), what changed (including the old and new value), when (including the time zone if applicable), and a reason for the change, which the system should prompt for.²

For extraction, the model is effectively one of the operators. Its entries need the same treatment as a person’s, plus the information that ties each value to the page. A complete record for each field includes:

Record the model and configuration version on every value. When a vendor update or a threshold change happens, you will need to know exactly which values were produced under which version, so you can decide whether any need to be checked again.

ALCOA+ Mapped to Extracted Values

A Simple Traceability Test

Include this as a test case in the validation: pick values at random in the target system and time how long it takes to show the source page with the field highlighted, the model version, the confidence, the checks, and the review history. If it takes more than a minute or two, or needs a database query, the traceability exists only in theory. It is a reasonable request to expect during an inspection, and a quick, clean answer does more for confidence in the tool than any accuracy statistic.

Keeping the Tool in a Validated State

Validation evidence describes the tool on the day of the test, against the documents in the test set. Documents change, suppliers change, scanners get replaced, and vendors update models. The operational controls decide whether the evidence stays true.

Monitor What You Can See and What You Cannot

Draft Annex 22 asks for the performance of a model, as defined by its metrics, to be monitored regularly, and for regular monitoring of whether input data is still within the model’s sample space and intended use, with metrics defined to detect drift in the input data.⁴

For extraction, part of the performance picture is easy to see. The correction rate on routed fields shows how often the tool is wrong on the values people check. The more important part is hidden: wrong values that were auto-accepted are not corrected by anyone, so they never appear in a correction log. The only way to measure them in production is to sample. Set up a periodic verification sample of auto-accepted Tier 1 values, independently re-keyed from the source, sized using the same arithmetic as the validation. That sample is the production version of your main acceptance metric.

Input drift for documents is usually concrete and easy to name: a new supplier, a supplier that changes its certificate template, a new master batch record version, a new site, a new scanner model, a new language. Track the share of documents in each known subgroup and flag documents that do not match any known layout. Those should go to full review until the layout has been assessed.

Change Control Triggers

Draft Annex 22 says the tested model, the system it runs in, and the whole process it supports should be under change control before deployment. Any change to the model, the system, or the process, including a change to physical objects the model uses as input, should be documented and evaluated to decide whether the model needs to be tested again, and a decision not to retest should be fully justified. The model should also be under configuration control, with measures to detect unauthorized change.⁴

For extraction tools, write the triggers into the change control procedure so nobody has to argue about them later:

• Any vendor model or engine update, including updates described as minor.
• A new version of a master batch record or logbook form that the tool reads.
• A new supplier, or a changed layout from an existing supplier.
• Replacement of scanners or capture devices, or changes to scan settings.
• Changes to confidence thresholds, rule checks, or normalization rules.
• Adding a field, a document type, or a site to the intended use.

Retesting after a change raises the independence question again. Draft Annex 22 asks that test data use be recorded, including how many times it was used.⁴ A test set used repeatedly to check changes slowly becomes familiar to the team. Refresh it with newly collected documents on a planned cycle.

Supplier Responsibilities

Most extraction tools are bought rather than built. Draft Annex 22 expects documentation of training, validation, and testing to be available to and reviewed by the regulated user, regardless of who did the work.⁴ The draft Annex 11 revision says the regulated user is fully accountable for qualification and validation documentation supplied by a vendor and should review it carefully before relying on it.²

The supplier agreement should cover the items your validation depends on: version pinning and advance notice of model changes, release notes that state whether the extraction model changed, a commitment not to train shared models on your documents without agreement, access to the vendor’s own test evidence, and export of audit trail data in a usable format.

Conclusion

Document extraction AI can remove a great deal of manual transcription from GMP operations and finally make paper-based data usable for trending and investigation. It earns that role only when the validation matches what the tool does: it writes data. That means validating field by field, against an independent set of real documents, with acceptance criteria tied to the process being replaced and to an honest count of how many errors would pass unreviewed. It means treating confidence thresholds, rule checks, and human review as one routing design to be tested, and keeping every value linked to the place on the page it came from. And while draft Annex 22 remains a draft, the type of model inside the tool matters, because the current text draws a firm line around generative models in critical applications.

Sakara Digital works with pharma and biotech organizations putting AI into regulated processes, including the step where paper becomes data. If you are assessing an extraction tool, or trying to decide what evidence your quality unit will need before it goes live, we are happy to have that conversation.

For Further Reading