In This Article
- Executive Summary
- Why Retraining Is the Change That Breaks Change Control
- What the Regulatory Documents Say, and What Status They Hold
- Six Retraining Cases That Are Not the Same Change
- The Decision Tree: Nine Steps from Trigger to Release
- What Evidence Survives a Retrain and What Must Be Regenerated
- Approval Authority and What the Release Record Must Contain
- Two Worked Examples: A QC Image Classifier and a Document Classifier
- Rollback, Parallel Running, and Writing the Rules Down in Advance
- Conclusion
- For Further Reading
- References & Sources
Executive Summary
Most pharma and biotech organizations that have put a machine learning model into a GxP process have written some form of AI change control procedure. Very few have written the part that actually gets used. The change that arrives most often is not a new use case or a new system. It is a retrain: the same model, the same intended use, new weights. And a generic change control procedure gives the team no way to decide whether that retrain is a paperwork exercise or a full revalidation.
The reason it is hard is that retraining is not one change. It is at least six. Retraining the same architecture on more of the same data is a different regulatory event from retraining on data from a new site, which is different again from swapping in a new foundation model version, which is different again from accepting an update a vendor pushed to you. Each one preserves a different portion of the original validation evidence and invalidates a different portion. Treating them all the same means either revalidating everything every quarter, which nobody sustains, or waving everything through, which nobody can defend.
This article gives a decision tree for that single decision. It classifies the six retraining cases against a like-for-like, minor, and major scale, sets out a nine-step path from trigger to release, and provides an evidence inventory showing which parts of the original validation package survive each type of retrain and which have to be regenerated. It works the tree through two examples, a QC image classification model and a document classification model, and closes with rollback, parallel running, and how to write the retraining rules down before the first retrain rather than after it.
Why Retraining Is the Change That Breaks Change Control
A conventional change control system is built around a stable assumption: the thing being changed is described by a specification, the change is a difference from that specification, and the assessment asks what the difference touches. Change a chromatography method parameter and you can point at the parameter. Change a line of configuration in a manufacturing execution system and you can point at the line. The assessment scopes itself.
A retrained model breaks that assumption in a specific way. Nothing in the specification changes. The intended use is the same, the input sample space is the same, the acceptance criteria are the same, the user interface is the same, the integration points are the same. What changed is a set of numeric parameters that no one can read and that the specification never described in the first place. The system behaves differently and the documentation says nothing has changed.
This is why so many retraining decisions in life sciences get made informally. A data science team notices performance sliding, adds three months of new images to the training pool, retrains, sees a better number on their internal test split, and asks quality to approve a like-for-like change. Quality has no basis to disagree and no basis to agree. The change record ends up saying something close to nothing, and the organization has quietly moved to a model that no one has independently tested.
The second reason retraining is hard is that the evidence supporting the original model was never organized in a way that makes partial reuse possible. A validation package written as one document, for one model, at one point in time, cannot answer the question “which of these sections still holds.” That question is the whole of retraining change control. If your package cannot answer it, every retrain looks like a full revalidation, and the practical result is that models do not get retrained at all. They sit in production, degrading, because updating them is more painful than tolerating the drift.
The failure mode worth naming. The most common outcome in organizations without a retraining decision tree is not an unsafe retrain. It is a model that is never retrained. Teams avoid the change control burden, performance decays, the human reviewers quietly start overriding the model more often, and the model becomes decorative. That is a data integrity problem, because the process description still says the model is performing a function it no longer performs reliably.
Retraining is a GxP event even when performance improves
There is a persistent belief that a retrain which improves a metric is self-justifying. It is not, for two reasons that matter to an inspector.
First, the improvement was measured on something. If the new test set is not independent of the new training data, the improvement is an artifact of measurement rather than a property of the model. Data leakage is not a hypothetical failure. A systematic survey of machine learning based science identified 17 fields with documented leakage errors, collectively affecting 329 papers, in some cases producing conclusions that did not survive reanalysis.15 Those were published papers under peer review. A retrain done under schedule pressure inside a company has fewer checks, not more.
Second, an aggregate improvement can hide a subgroup regression. A model that gains two points of overall accuracy while losing sensitivity on a rare but critical defect class has become worse at the thing it exists to do. Aggregate metrics are the wrong unit of analysis for a GxP acceptance decision, and the draft EU GMP Annex 22 is explicit that acceptance criteria may differ for specific subgroups within the intended use.1
What the Regulatory Documents Say, and What Status They Hold
Before building a decision tree, it is worth being precise about what is actually in force, what is draft, and what applies to a different product category entirely. Getting this wrong in a validation rationale is an easy finding for an inspector to write.
Draft EU GMP Annex 22 is still a draft
The European Commission released a draft Annex 22 on artificial intelligence for public consultation in July 2025, alongside a revised draft Annex 11 on computerised systems and a revised draft of Chapter 4 on documentation.14 The consultation closed on 7 October 2025. As of September 2026 there is no adopted final text and no published implementation date. EMA’s GMP and GDP inspectors working group has continued to work the file, including a multistakeholder workshop in mid-2026 on how generative models might be handled, and the HMA and EMA AI group has continued meeting with industry.18 Nobody should be writing a procedure that claims Annex 22 compliance as a current regulatory obligation. It is entirely reasonable to design toward it, and we think teams should, but the status has to be stated accurately in the document.
What the draft says about change is short and unusually direct. Section 10.1 places the tested model, the system it runs in, and the whole process it automates or assists under change control before deployment. It then requires that any change to the model, the system, or the process, including any change to the physical objects the model uses as input, be documented and evaluated to determine whether the model needs to be retested. A decision not to retest has to be fully justified.1
That single clause is the legal shape of the decision tree. The draft does not tell you when to retest. It tells you that you must decide, in writing, with a rationale, every time. The tree in this article is a way of making that decision reproducible rather than personal.
A scope point that changes everything about retraining. The draft annex applies only to static models, meaning models that do not adapt during use by incorporating new data. It states that dynamic models which continuously and automatically learn during use are not covered and should not be used in critical GMP applications. It also excludes models with probabilistic output, and therefore generative AI and large language models, from critical GMP applications.1 The practical consequence is that under this draft there is no continuous learning path in critical GMP. Every model update is a discrete, dated, documented event with a before and an after. That is a constraint, but it is also what makes a decision tree possible at all.
The joint FDA and EMA principles put lifecycle management on the record
In January 2026 EMA and FDA published a joint set of ten guiding principles of good AI practice in drug development, covering AI used to generate or analyze evidence across nonclinical, clinical, post-marketing, and manufacturing phases.23 These are principles, not requirements, and both agencies said they are not prescriptive. They still matter, because they are the clearest signal yet of what two major regulators expect a lifecycle argument to look like.
Principle 9 is the relevant one. It calls for risk-based quality management systems implemented throughout the AI technology’s life cycle, including support for capturing, assessing, and addressing issues, and it says the technologies undergo scheduled monitoring and periodic re-evaluation to ensure adequate performance, giving data drift as the example.2 Principle 2 sets the risk-based approach with proportionate validation. Principle 10 asks for clear, accessible information to the intended audience about the technology’s context of use, performance, limitations, underlying data, and updates.
Read together, those three principles say that a retrain is expected, that its depth of assessment should be proportionate to risk, and that the people relying on the model are entitled to know it happened. Most retraining procedures we see satisfy the first, gesture at the second, and forget the third entirely.
ICH Q10 supplies the change management system the tree plugs into
None of this replaces the pharmaceutical quality system. ICH Q10 already defines a change management system as one of the four elements of a pharmaceutical quality system. Section 3.2.3 requires that quality risk management be used to evaluate proposed changes with the level of effort and formality commensurate with the level of risk, that proposed changes be evaluated by expert teams contributing appropriate expertise from relevant areas, that prospective evaluation criteria for a proposed change be set, and that after implementation an evaluation be undertaken to confirm the change objectives were achieved and that there was no deleterious impact on product quality.10
Two of those four are the parts most AI retraining procedures skip. Setting prospective evaluation criteria means the acceptance criteria for the retrained model are agreed before the model exists, which is exactly what stops a team from choosing the metric that makes the candidate look best. And the post-implementation evaluation means a retrain is not finished when the model is released: Q10 expects a look back to confirm the change did what it was supposed to do. For a model, that means a defined post-release monitoring window with pre-agreed metrics, not a general intention to keep an eye on it.
FDA’s PCCP is a useful analogy and it is device guidance, not drug guidance
Be accurate about this one. FDA’s final guidance on Predetermined Change Control Plans, issued in December 2024, is titled “Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions.”67 It applies to medical devices and their marketing submissions. It does not apply to an AI model used in drug substance or drug product manufacturing, and citing it as an authority for a GMP model would be a category error. It is worth studying as a structure, not as a requirement.
The structure is the useful part. A PCCP has three components: a description of the planned modifications, a modification protocol setting out the methods used to develop, validate, and implement those modifications, and an impact assessment. The final guidance broadened scope from machine learning specifically to all AI-enabled device software functions, and added recommendations that manufacturers consider the characteristics of their intended use populations and environments.6
What a PCCP does conceptually is move the regulatory conversation forward in time. Instead of assessing each change when it happens, the sponsor agrees in advance on the envelope of changes that are permitted and the protocol by which they will be qualified. Anything inside the envelope proceeds under the protocol. Anything outside it is a new conversation.
The drug side has had that concept for years under different names. ICH Q12 provides the post-approval change management protocol, a regulator-agreed protocol that defines the proposed change, the supporting studies, the acceptance criteria, and the intended reporting category before the change is made.11 FDA’s 2022 final guidance on comparability protocols does the same thing for CMC postapproval changes, describing a comparability protocol as a comprehensive, prospectively written plan for assessing the effect of a proposed change.12
The pattern in all three is identical, and it is the pattern a retraining procedure should follow: decide the rules before the change, so that the change itself becomes an execution rather than a negotiation.
GAMP and the international device principles fill in the method
GAMP 5 second edition supplies the risk-based validation approach and the critical thinking framing that a proportionate retraining assessment depends on, and ISPE’s GAMP Guide on Artificial Intelligence, published in July 2025, extends that into model-specific territory. ISPE’s own formal comments on the draft Annex 22, submitted at the close of the consultation, are a good read on where industry thinks the draft is workable and where it is not.5
Two of the ten Good Machine Learning Practice guiding principles published by FDA, Health Canada, and MHRA in October 2021 speak directly to retraining. Principle 4 is that training data sets are independent of test sets. Principle 10 is that deployed models are monitored for performance and re-training risks are managed.8 Those principles were written for devices, but principle 4 in particular is a statement about the validity of measurement that does not change when the product category changes. The draft Annex 22 arrives at the same requirement through its own route, and goes further, requiring procedural or technical controls to prevent staff who have seen the test data from working on training and validation of the same model, with a four-eyes arrangement where that separation is impossible.1
Six Retraining Cases That Are Not the Same Change
The first job of the tree is to identify which retrain you are actually doing. In practice, teams describe every one of these as “we retrained the model,” and the description hides the differences that determine the assessment.
Same architecture, more of the same data
The model structure, hyperparameters, and preprocessing are unchanged. The training pool has grown with data drawn from the same sites, instruments, and populations already characterized in the intended use description.
New data from a new source
Training data now includes a new manufacturing site, a new instrument or camera, a new product presentation, or a population not represented before. The input sample space has moved.
New hyperparameters or new preprocessing
Same architecture and same data, but a changed learning rate, class weighting, augmentation policy, decision threshold, or normalization step. The model family is the same; the model is not.
New architecture or new foundation model version
A different network structure, a different algorithm family, or a new version of an underlying pretrained model whose weights you do not control and cannot inspect.
Vendor-pushed update
The model is embedded in a supplier’s product and the supplier has updated it, sometimes with limited notice and limited disclosure of what changed.
Scheduled periodic retrain versus drift-triggered retrain
This is a modifier on any of the cases above, not a case in itself. It changes the trigger evidence, the monitoring obligations, and the approval path, and it is the distinction most procedures omit.
Classifying the cases
A three-level classification is enough for most organizations, and more levels make the tree harder to apply consistently. We use like-for-like, minor, and major. The classification is a starting position, not a conclusion: any case escalates a level if the model supports a batch release decision, a product disposition, or a regulatory submission, and any case escalates if the post-retrain test shows a subgroup regression.
| Case | Starting classification | What triggers revalidation | Approval level |
|---|---|---|---|
| A. Same architecture, more of the same data | Like-for-like, if and only if the input sample space is demonstrably unchanged | Any subgroup falling below its acceptance criterion; any change to the held-out test set; failure to beat the incumbent model on the fixed benchmark | System owner plus QA reviewer; no change control board |
| B. New data from a new site, instrument, or population | Major | Automatic. The intended use description and input sample space characterization both change, which invalidates the original test set’s claim to representativeness | Change control board with process SME, QA, and data science |
| C. New hyperparameters or preprocessing | Minor | Any change to a decision threshold, class weighting, or preprocessing step that alters the operating point; any change in the confidence score distribution | System owner plus QA approval, with data science sign-off on the parameter rationale |
| D. New architecture or new foundation model version | Major | Automatic. Explainability evidence, failure mode analysis, and robustness testing are all architecture-specific and do not transfer | Change control board, with a documented decision on whether this is a new model rather than a change to an existing one |
| E. Vendor-pushed update | Major until proven otherwise | Automatic, unless the supplier agreement gives you advance notice, a change description sufficient to classify it into cases A through D, and the ability to defer installation | Change control board plus supplier quality; escalate to the quality agreement if notice was not given |
| F. Drift-triggered rather than scheduled | Escalates the underlying case by one level | The drift itself is a deviation trigger before it is a change trigger. Investigate why the input distribution moved before deciding to retrain against it | As per the underlying case, plus deviation and CAPA linkage |
Why the scheduled versus drift-triggered distinction matters more than it looks
A scheduled retrain is a maintenance activity. A drift-triggered retrain is a response to something having gone wrong in the process, the instrument, the material, or the population. The retrain may still be the right response, but retraining against drift without investigating the drift means the model absorbs a process change that nobody has explained. If a camera’s illumination has degraded, retraining the classifier on the degraded images makes the model tolerant of a broken camera. That is not a model improvement. It is a masked equipment problem.
The draft Annex 22 anticipates this by requiring two separate monitoring streams: system performance monitoring against the defined metrics, with degradation or change of lighting conditions given as an explicit example, and input sample space monitoring with defined metrics for drift in the input data.1 Those two streams answer different questions. Performance monitoring asks whether the model is still right. Input monitoring asks whether the world it was built for still exists.
The clinical machine learning literature is instructive on how much difference the trigger makes. A study of 143,049 adult inpatients across seven hospitals, published in JAMA Network Open in 2025, built a label-agnostic monitoring pipeline to detect harmful data shifts without waiting for outcome labels. It found significant shifts driven by demographic changes, hospital type, admission source, and changes to critical laboratory assays. Transfer learning improved performance at community hospitals by a modest margin, while drift-triggered continual learning during the COVID-19 period produced a far larger recovery in discrimination.13 The point for a GxP setting is not the specific numbers. It is that the retraining strategy which works depends on what caused the shift, which means you have to know what caused it before you choose.
The Decision Tree: Nine Steps from Trigger to Release
This is the operating procedure. It is written as an ordered sequence because the order matters: several steps exist specifically to stop the team from moving forward before a question has been answered. In practice these steps map onto a change request form with sections that cannot be signed out of order.
Record the trigger and classify it before touching the model
Write down what prompted the retrain: a calendar date, a monitoring alarm on model performance, a monitoring alarm on input drift, a process or equipment change, a supplier notification, or a deviation. If the trigger is a drift or performance alarm, open the deviation first. The retrain is a potential corrective action, and it cannot be evaluated as one until the cause is understood. Nothing else in this tree proceeds while the trigger is unclassified.
Ask whether the intended use or the input sample space has moved
Compare the current process against the approved intended use description and the characterization of the input sample space, including the defined subgroups. New site, new instrument, new product presentation, new defect class, new document type, new language, new population: any of these is a change to the sample space, which puts you in case B and makes the change major regardless of how routine the retrain feels. If the sample space is unchanged and you can show it, you are in case A, C, or D depending on what you are changing about the model.
Confirm the test set is still independent, and that it is still the right test set
This is the step most likely to be skipped and most likely to invalidate everything after it. Confirm that no record now in the training pool has ever been in the held-out test set, that no near-duplicate of a test record has entered training, and that the staff who prepared the new training data have not had access to the test set. Where test data comes from physical objects, confirm the objects used for the final test have not been used to train or validate. Then ask the harder question: given the new training data, is the old test set still representative of the intended use, or has it become a historical artifact? A test set that no longer reflects current inputs will pass a model that is about to fail in production.
Fix the benchmark set before the new model exists
Nominate a frozen benchmark set, versioned and access-controlled, on which both the incumbent model and the candidate model will be scored. This is separate from the acceptance test set and it exists to answer one question that acceptance criteria cannot: is the new model better than the one we already trust, on identical inputs? Record which data has been used for testing, when, and how many times, so the benchmark’s own exposure is traceable. A benchmark chosen after seeing the candidate’s results is not a benchmark.
Write the retraining protocol and get it approved before training starts
The protocol states the case classification, the training data inclusion and exclusion rules with justification, the preprocessing specification, the hyperparameters that may vary and the ranges they may vary within, the acceptance criteria for every metric and every subgroup, the comparison rule against the incumbent on the benchmark set, the bias and subgroup checks, the explainability artifacts required, and the definition of a failed run. A process subject matter expert approves the acceptance criteria. This protocol is the retraining equivalent of a comparability protocol or a post-approval change management protocol, and it earns its keep the same way: the argument happens once, in advance, rather than every time under pressure.
Train, then test once against the protocol
The candidate model is evaluated against the held-out test set a single time, following the approved script. Repeated evaluation against the same test set is a slow form of leakage: each look leaks a little information into the selection of the model. If the model fails, that is a documented failed run, and the next attempt uses fresh test data or an explicitly justified reuse. Any deviation from the test plan, any failure to meet acceptance criteria, and any omission of test data is documented, investigated, and justified.
Run the four comparisons, not just the one
Aggregate performance against the original acceptance criteria. Subgroup performance against subgroup criteria, every subgroup, no exceptions. Head-to-head against the incumbent model on the frozen benchmark, reported as a difference with an indication of uncertainty rather than two isolated numbers. And a disagreement analysis: on which records do the two models differ, and does a subject matter expert think the new model is right on those records? The disagreement analysis catches things all three metric comparisons miss, because it puts a human in front of the specific cases where behavior changed.
Refresh the confidence and explainability evidence to the extent the case requires
Confidence score distributions shift after retraining, which means a threshold that was correctly set for the old model may be wrong for the new one. Recheck the threshold and the rule for flagging low-confidence outputs as undecided. For explainability, regenerate feature attribution on a defined sample and confirm the features driving decisions are still the ones a subject matter expert considers relevant. For cases A and C a targeted refresh on the same sample is usually enough. For cases B, D, and E the explainability evidence has to be rebuilt, because the features the model relies on can move even when the metrics do not.
Release, then close the loop after implementation
Approve at the level the classification requires, place the new model under configuration control with a detection mechanism for unauthorized change, notify the users that the model has been updated and what changed about it, and define the post-release monitoring window with the metrics that will be reviewed and the date the review happens. ICH Q10 expects change effectiveness to be evaluated after implementation to confirm the objectives were achieved and no unintended harmful impact occurred. Put that review on the calendar as part of the change record, not as an intention.
The one question that decides most retrains
If you strip the tree back to a single question, it is step 2: has the input sample space moved? If it has not, the retrain is a controlled update within a validated envelope and the evidence largely survives. If it has, the original test set no longer represents the intended use, the intended use description itself needs revision, and you are doing something much closer to a new validation than a change. Teams that get this one question right get most of the rest right by consequence.
What Evidence Survives a Retrain and What Must Be Regenerated
This is the table that makes proportionate assessment possible. It only works if the original validation package was structured so that its parts can be cited individually. If your package is a single monolithic report, the first improvement to make is not a retraining procedure. It is breaking that report into addressable sections with their own version numbers.
| Evidence artifact | Case A: same data source | Case B: new source | Case C: hyperparameters | Case D: new architecture | Case E: vendor update |
|---|---|---|---|---|---|
| Intended use description and input sample space characterization | Survives; confirm in writing | Must be revised and re-approved | Survives | Survives if the use is unchanged | Must be reconfirmed against the vendor’s change note |
| Subgroup definitions | Survives | Must be extended for the new source | Survives | Survives | Survives, but must be tested again |
| Acceptance criteria and their rationale | Survives | Survives; new subgroups need criteria | Survives | Survives | Survives |
| Test data selection rationale and labeling verification | Survives if the set is still representative | Regenerate; the set is no longer representative | Survives | Survives | Survives |
| Test data independence controls and access records | Must be re-evidenced for the new training pool | Regenerate | Must be re-evidenced | Must be re-evidenced | Unobtainable for vendor internals; substitute your own independent acceptance test |
| Test execution results (aggregate and subgroup) | Regenerate | Regenerate | Regenerate | Regenerate | Regenerate |
| Comparison to the prior model on the frozen benchmark | Required | Required, plus a benchmark extension covering the new source | Required | Required | Required, and this is often the only evidence you can generate |
| Bias and subgroup fairness checks | Targeted rerun on defined subgroups | Full rerun; the new source is a new bias surface | Targeted rerun | Full rerun | Full rerun on your own data |
| Explainability and feature attribution evidence | Targeted refresh on the same sample | Rebuild | Targeted refresh | Rebuild; the technique itself may need to change | Rebuild to whatever depth the vendor’s disclosure allows |
| Confidence score threshold and undecided rule | Recheck | Reset and re-justify | Reset and re-justify | Reset and re-justify | Reset and re-justify |
| Infrastructure and system qualification (IQ/OQ) | Survives | Survives | Survives | Reassess; new architecture may change the runtime | Reassess against the vendor’s release notes |
| Process integration, human-in-the-loop procedure, and user training | Survives; notify users of the update | Reassess; new source may change who reviews what | Reassess if the operating point moved | Reassess | Reassess |
| Supplier assessment and quality agreement | Not applicable in-house | Not applicable in-house | Not applicable in-house | Applies if the base model is external | Review; a surprise push is a supplier quality issue in its own right |
The rows people argue about
Three rows generate most of the discussion in a review meeting.
Test data independence controls, case A. Teams want this to survive because nothing about the test set changed. But independence is a property of the relationship between two datasets, and one of them just changed. The evidence that has to be produced is a fresh check that the enlarged training pool does not intersect the test set, including near-duplicates. In an image application this means checking that the new images are not additional photographs of objects already represented in the test set. The draft annex makes this explicit for physical objects: objects used for the final test should not previously have been used to train or validate, unless features are independent.1
Explainability evidence, case C. A hyperparameter change feels too small to disturb which features the model relies on. Sometimes that is true. Class weighting changes and augmentation policy changes are the exceptions, because both deliberately alter what the model pays attention to. The cheap protection is to keep a small fixed explainability sample, twenty to fifty records, and regenerate attribution on it after every retrain of any type. It takes an afternoon and it catches the case where the metrics held steady while the model started reading a different part of the image.
Confidence thresholds, everywhere. Thresholds are the most commonly forgotten artifact. A retrained model is usually better calibrated in some regions of its output range and worse in others, so a threshold tuned for the old model can push more cases into automated decisions than intended or fewer. Any change to the threshold is itself a change to the operating point and should be recorded as part of the change, not as a configuration detail.
Approval Authority and What the Release Record Must Contain
Classification without differentiated approval authority achieves nothing. If every retrain goes to the same board, the board becomes a rubber stamp for the routine cases and has no capacity left for the ones that matter.
Who approves what
- Like-for-like (case A with an unchanged sample space). System owner approves. QA reviews the change record and the test results, with authority to escalate. The process subject matter expert confirms in writing that the sample space is unchanged. No board.
- Minor (case C). System owner and QA approve jointly. Data science signs the parameter rationale. The process subject matter expert approves any change to the operating point, because that is a change to how the process behaves, not to how the model is built.
- Major (cases B, D, E, and anything escalated). Change control board including the process subject matter expert, QA, data science, IT, and the system owner. For case E, supplier quality attends. The board’s specific job is to decide whether the change is a change at all or the introduction of a new model that needs its own validation identity.
- Any case with a subgroup regression. Escalates to the board irrespective of starting classification, and the board decides whether to accept the regression with a documented rationale, reject the candidate, or accept it with a compensating control such as increased human review on the affected subgroup.
The approval role people forget
The draft Annex 22 assigns responsibility for the intended use description and for the acceptance criteria to a process subject matter expert, not to the data science function and not to QA.1 That allocation is deliberate and it should carry into the retraining procedure. The person who owns whether a defect classification is correct is the person who knows the product and the process, not the person who trained the model. Retraining approvals that route around the process expert are the ones that produce a technically valid model doing the wrong job well.
The release record
The release record is the document an inspector will ask for, and it should be readable on its own without requiring someone to reconstruct the story from a dozen systems. At minimum it contains:
- Model identity. A unique version identifier for the model artifact, the identifier of the model it replaces, a cryptographic hash of the deployed artifact, and the date and time it entered production.
- Case classification and rationale. Which of the six cases, why, and the written confirmation of whether the input sample space moved.
- Trigger record. What prompted the retrain, with a link to the deviation if the trigger was a drift or performance alarm.
- Training data manifest. The dataset version, record counts by subgroup, the date range covered, inclusion and exclusion rules applied, and the justification for any exclusion.
- Independence attestation. The evidence that test data did not enter training, the access control and audit trail records for the test repository, and the names of staff on each side of the separation.
- Approved protocol reference and any deviations from it.
- Test results. Aggregate and per-subgroup metrics against acceptance criteria, with pass or fail stated per criterion rather than summarized.
- Benchmark comparison. The frozen benchmark version, the incumbent’s score, the candidate’s score, the difference, and the disagreement analysis with the subject matter expert’s conclusions on the disputed records.
- Confidence and explainability artifacts. The threshold before and after, the undecided rule, and the attribution evidence at whatever depth the case required.
- Approvals. Named individuals, roles, and dates, matching the classification’s required approval level.
- Rollback plan. The retained prior artifact’s location, the trigger conditions for reverting, the technical steps, and the named decision-maker.
- Post-implementation review. The metrics, the window, the scheduled date, and later the outcome.
- User notification. What was communicated to the people who rely on the model’s output, and when.
That list looks long. In practice it fits on four pages and most of it is populated automatically from the model registry and the training pipeline. The effort is in building the record structure once, not in filling it in each time.
Two Worked Examples: A QC Image Classifier and a Document Classifier
The tree is easier to judge against concrete cases. Both examples below are constructed illustrations, not accounts of any particular organization’s system.
Example one: a QC image classification model for visual inspection
A model classifies images of filled vials into accept and reject categories, with reject subdivided into defect types: particulate, cosmetic, fill volume, and closure. It runs on an inspection line, output feeds a human reviewer who confirms rejects, and the model’s role in the process description is to reduce the volume of images requiring human review while maintaining the line’s established defect detection performance. Acceptance criteria are set per defect type, with the tightest criterion on particulate because that is the safety-relevant class.
The trigger. Monthly input monitoring shows a shift in the image brightness distribution starting eleven weeks ago. Model performance monitoring shows overall accuracy stable but a two-point decline in sensitivity on cosmetic defects.
Step 1. This is a drift-triggered retrain, case F modifier. A deviation is opened before any retraining work begins. The investigation finds a partial degradation in one of the inspection station’s LED arrays. That finding changes everything downstream: the correct first action is to repair the illumination, not to retrain the model to tolerate a failing light. The engineering fix is made. The model is then re-evaluated on post-repair images and found to be performing within criteria again.
What actually happens next. The deviation closes with an equipment CAPA and an added preventive control: illumination intensity is added to the input monitoring metrics with an alert threshold set below the level at which model performance was affected. No retrain occurs. This is the outcome a good decision tree produces most often, and it is the one that never appears in vendor material about model lifecycle management.
A second trigger, three months later. A new vial presentation is introduced on the same line, with a different glass supplier and a slightly different shoulder profile. Engineering asks for the model to be retrained to cover it.
Steps 2 through 4. The input sample space has moved: this is case B, major, with no argument available. The intended use description is revised to add the new presentation, and the subgroup structure is extended so that the new presentation is a subgroup within each defect type. The existing test set contains no images of the new vial, so it cannot demonstrate performance on it. New test images are captured after the training set is closed, using physical vials that have never been used in training, and are labeled by two independent inspectors with disagreements adjudicated by a third. The frozen benchmark is extended with a stratified sample covering both presentations so that the incumbent and candidate can be compared on the old presentation as well as the new one.
Steps 5 through 7. The protocol sets acceptance criteria per defect type per presentation. The candidate passes aggregate criteria and passes on the new presentation. On the frozen benchmark, it matches the incumbent on the old presentation for three defect types and loses one point of sensitivity on particulate for the old presentation. That is a subgroup regression on the safety-critical class.
What the board does with that. This is where the tree earns its existence. The regression is small, the aggregate numbers look good, and the pressure to release is real because the new presentation is on the production schedule. The board’s options are documented: reject the candidate and retrain with rebalanced class weighting for particulate on the old presentation, accept with a compensating control routing all old-presentation particulate borderline cases to human review, or accept the regression with a written risk rationale. Without the frozen benchmark and the per-subgroup criteria, none of these options would have been visible, and the model would have been released on its aggregate improvement.
What made this work. Three decisions taken long before the retrain: acceptance criteria defined per defect type rather than in aggregate, physical test objects segregated from training objects, and a frozen benchmark held for model-to-model comparison. None of them are expensive. All of them have to exist before the change arrives, because none can be constructed retrospectively without compromising independence.
Example two: a document classification model in regulatory operations
A model classifies incoming documents into a controlled taxonomy for a document management system: clinical study report, investigator brochure, site correspondence, regulatory authority correspondence, and so on. Misclassification does not directly affect product quality, but it affects retrievability, which affects inspection readiness and submission assembly. The model runs with a confidence threshold: above it, the classification is applied automatically; below it, the document is routed to a records specialist.
The trigger. The vendor supplying the classification component announces that the underlying pretrained language model is being upgraded to a new version, and that the old version will be retired in ninety days.
Step 1 and 2. This is case E crossed with case D: a vendor-pushed update that is also a change of underlying architecture. Both classify as major. The intended use has not changed and the document types have not changed, so the sample space is stable, but that provides no comfort here because the model’s internals have been replaced entirely.
The supplier conversation. The first action is not technical. It is to establish what the supplier will disclose: what changed, whether the previous version can be pinned for a defined period, whether the supplier ran any comparative evaluation, and what the quality agreement says about notification periods for model changes. If the answer is that the change is mandatory in ninety days with no comparative data, that is a supplier quality finding, and it should be recorded as one regardless of how the technical assessment turns out. A regulated user cannot accept a change to a GxP-relevant function on a timeline that does not permit assessment. The draft annex is clear that documentation for these activities should be available and reviewed by the regulated user whether the model was built in-house or supplied.1
Steps 3 and 4. Test data independence takes a different form here. You cannot verify that your test documents were excluded from the supplier’s pretraining corpus, and for a general-purpose pretrained model you should assume public documents may have been included. The response is to build the acceptance test set from documents that are internal and were never public: internal correspondence, internal study documents, site communications. Those cannot have been in the vendor’s pretraining data. The frozen benchmark is a stratified sample across all document types, including the low-frequency types that carry the most retrieval risk.
Steps 5 through 8. The protocol specifies per-class acceptance criteria, with tighter criteria on regulatory authority correspondence than on general site correspondence because misfiling the former has a larger consequence. Testing shows the new version improves on most classes and degrades on one: it now confuses two adjacent document types that the previous version separated cleanly. The confidence distribution has also shifted upward across the board, meaning the existing threshold would send fewer documents to human review than before, including some of the newly confused ones. The threshold has to be reset, and reset on data, not by preference.
The release decision. The board accepts the new version with three conditions: the threshold is lowered for the two confused classes specifically, creating a class-conditional threshold rather than a single global one; the records specialists are briefed on the specific confusion so they know what to look for; and the post-implementation review at sixty days examines the misclassification rate on those two classes specifically rather than reporting an overall number. The user notification is not optional here. People who have learned to trust the model’s behavior are entitled to know that its behavior changed, which is what the joint FDA and EMA principle on clear, essential information is asking for.2
Rollback, Parallel Running, and Writing the Rules Down in Advance
Rollback is a control, not a contingency
A model rollback is easier than most software rollbacks and harder than teams assume. Easier, because the artifact is a file and reverting to the previous file is mechanically simple. Harder, because of everything around the file.
Four things have to be retained for a rollback to be real:
- The exact prior artifact, with its hash, held under configuration control for a defined retention period rather than on whatever storage the last deployment happened to use.
- The prior artifact’s full runtime dependency set. A model file is useless without the library versions, preprocessing code, and configuration it was tested with. Reverting the model and not the preprocessing produces a system that was never validated in that combination.
- The prior thresholds and configuration. If thresholds moved with the retrain, rolling back the model without rolling back the thresholds produces a third configuration nobody has tested.
- The records produced under the new model. Every output the new model produced between release and rollback needs to be identifiable, because a rollback implies that those outputs may be suspect. Which means every record has to carry the model version that produced it. If your records do not carry a model version, you cannot roll back cleanly, and you should fix that before you fix anything else in your retraining procedure.
The rollback trigger conditions belong in the change record, agreed before release. Useful triggers are concrete: performance on a defined metric falling below a stated value over a stated window, the human override rate rising above a stated level, or a confirmed misclassification on a safety-relevant subgroup. A trigger that says “if problems are observed” is not a trigger.
Rollback is itself a change. Reverting to the previous model is a change to the system and goes through change control, on an expedited path if you have defined one. What makes it fast is that the previous model’s validation evidence is intact and current, so the assessment is short. What makes it slow, in organizations that have not thought about it, is discovering during an incident that the expedited path does not exist and the standard path takes three weeks.
Parallel running: what it proves and what it does not
Running the candidate model alongside the incumbent in production, with only the incumbent’s output acted upon, is the strongest evidence available for a major change. It is also frequently misunderstood.
What a parallel run proves. It shows how the two models behave on genuinely current production inputs, which is the population your test set can only approximate. It surfaces disagreements at a volume no curated test set reaches. It exposes throughput, latency, and integration problems before they matter. And it gives you a real distribution of confidence scores under production conditions, which is the only sound basis for setting a threshold.
What a parallel run does not prove. It is not a substitute for the independent acceptance test, because production inputs are unlabeled at the moment they arrive. A disagreement between two models tells you they differ; it does not tell you which is right. That determination requires adjudication by a subject matter expert on a sample of the disagreements, and that adjudication is the actual evidence. A parallel run reported as a percentage agreement, with no adjudication, is a number without meaning: two models can agree ninety-eight percent of the time and differ precisely on the cases that matter.
How to size it. Long enough to cover the process’s natural cycles. For a manufacturing model that means multiple batches, multiple shifts, and ideally more than one raw material lot. For a document model it means a period long enough to include the low-frequency document types. A two-week parallel run on a process with a monthly cycle proves less than it appears to.
A caution specific to GxP. The candidate model running in production is running in a production environment, and its outputs are being recorded. Decide in advance whether those shadow outputs are GxP records. Our position is that they should be treated as such, because they are being used to support a validation decision, which means they need the same integrity controls as any other record supporting a quality decision. Treating them as informal test output and then citing them in a validation report is the kind of inconsistency that does not survive an audit.
Write the rules before you need them
The final point is the one that determines whether any of this is usable. Every framework discussed here, the PCCP structure in device regulation, the post-approval change management protocol in ICH Q12, the comparability protocol in FDA’s CMC guidance, does the same thing: it moves the argument to before the change.61112
Applied to model retraining inside a pharmaceutical quality system, that means writing a retraining plan at the time the model is first validated, as part of the original validation package. The plan states, for that specific model:
- The retraining triggers, both scheduled and threshold-based, with the thresholds stated numerically.
- The changes permitted without escalation, which is to say the definition of like-for-like for this model, written specifically enough that two different people would classify the same change the same way.
- The changes that automatically escalate, including any change to the input sample space.
- The acceptance criteria and subgroup criteria that any future model must meet, so they are not renegotiated under pressure.
- The benchmark set identity and its governance.
- The independence controls and how they will be evidenced each time.
- The approval matrix by classification.
- The rollback plan and its trigger conditions.
None of this requires a regulator to agree in advance, which is the difference from a formal PCCP or PACMP. It is an internal commitment, held in the quality system, that makes each subsequent retrain an execution of an approved plan rather than a fresh negotiation. That is what turns a decision tree from a diagram in an article into something a team can actually run on a Tuesday afternoon when the monitoring dashboard turns amber.
A reasonable place to start if you have none of this
- Add a model version identifier to every record your models produce. Without it, rollback and investigation are both guesswork.
- Freeze a benchmark set for each production model, version it, and put it under access control. Do this before the next retrain, not during it.
- Restructure the validation package for one model into addressable sections so that partial reuse becomes possible. Use what you learn to set the template.
- Write the like-for-like definition for that one model. If two colleagues classify the same hypothetical change differently, the definition is not specific enough yet.
- Separate your monitoring into performance monitoring and input drift monitoring, with different owners and different alert routes. They answer different questions and should not share a dashboard tile.
Conclusion
Retraining is the change that exposes whether an AI governance framework was written to be used or written to be shown. A framework that treats every retrain identically will either exhaust the organization or wave through changes nobody assessed. The alternative is not more procedure. It is a small number of decisions taken early: define the input sample space precisely enough that you can tell when it moves, set acceptance criteria per subgroup rather than in aggregate, hold a frozen benchmark that no one trains against, keep the test data genuinely independent and be able to show it, and write down what counts as like-for-like for each specific model before the first retrain arrives.
The regulatory picture is going to keep moving. Annex 22 is still a draft with no final text and no implementation date, the joint FDA and EMA principles are deliberately non-prescriptive, and the PCCP framework that offers the cleanest structural analogy belongs to a different product category. None of that is a reason to wait. Every element of the decision tree in this article is defensible today under ICH Q10 change management and a risk-based validation approach, and every element of it will still be defensible when the final texts land. Organizations that build this now will be adjusting a working system rather than starting one.
Sakara Digital works with pharma and biotech organizations putting AI models into GxP processes and keeping them there through the changes that follow. If you are working out how to classify retraining changes, or you have a validated model you have been reluctant to update because the change control path is unclear, we are happy to have that conversation.
For Further Reading
For Further Reading
- How to Build an AI Change Control Process in Regulated Systems
- Building an AI Model Registry: What to Track and Why
- The AI Model Risk Assessment for Pharma: A Structured Checklist
- AI Incident Management in GxP: Classifying, Investigating, and CAPA-ing a Model Failure
- Computer System Validation vs. AI Validation: Key Differences That Matter
- Risk-Based AI Validation in GxP Environments: A Practical Guide
References & Sources
- European Commission. “Annex 22: Artificial Intelligence.” EudraLex Volume 4, draft for public consultation, July 2025. health.ec.europa.eu draft Annex 22 consultation guideline (PDF)
- European Medicines Agency and U.S. Food and Drug Administration. “Guiding principles of good AI practice in drug development.” January 2026. https://www.ema.europa.eu/en/documents/other/guiding-principles-good-ai-practice-drug-development_en.pdf
- Regulatory Affairs Professionals Society. “EMA, FDA issue joint AI guiding principles for drug developers.” January 2026. https://www.raps.org/news-and-articles/news-articles/2026/1/ema-fda-issue-joint-ai-guiding-principles-for-drug
- European Commission. “Annex 11: Computerised Systems.” EudraLex Volume 4, draft for public consultation, July 2025. health.ec.europa.eu draft Annex 11 consultation guideline (PDF)
- International Society for Pharmaceutical Engineering. “ISPE Comments: Annex 22 Artificial Intelligence, EMA and PIC/S consultation.” 7 October 2025. ISPE formal comments on draft Annex 22 (PDF)
- U.S. Food and Drug Administration. “Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions.” Final guidance, December 2024. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/marketing-submission-recommendations-predetermined-change-control-plan-artificial-intelligence
- U.S. Food and Drug Administration. Notice of availability, Federal Register Vol. 89, No. 233, 4 December 2024. https://www.govinfo.gov/content/pkg/FR-2024-12-04/html/2024-28361.htm
- Health Canada, U.S. Food and Drug Administration, and Medicines and Healthcare products Regulatory Agency. “Good Machine Learning Practice for Medical Device Development: Guiding Principles.” October 2021. https://www.canada.ca/en/health-canada/services/drugs-health-products/medical-devices/good-machine-learning-practice-medical-device-development.html
- U.S. Food and Drug Administration, Health Canada, and Medicines and Healthcare products Regulatory Agency. “Transparency for Machine Learning-Enabled Medical Devices: Guiding Principles.” June 2024. assets.publishing.service.gov.uk transparency guiding principles (PDF)
- International Council for Harmonisation. “ICH Harmonised Tripartite Guideline Q10: Pharmaceutical Quality System.” Step 4, June 2008. See section 3.2.3, Change Management System. database.ich.org ICH Q10 guideline (PDF)
- European Medicines Agency. “ICH guideline Q12 on technical and regulatory considerations for pharmaceutical product lifecycle management, Step 5.” ema.europa.eu ICH Q12 Step 5 guideline (PDF)
- U.S. Food and Drug Administration. “Comparability Protocols for Postapproval Changes to the Chemistry, Manufacturing, and Controls Information in an NDA, ANDA, or BLA.” Final guidance, October 2022. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/comparability-protocols-postapproval-changes-chemistry-manufacturing-and-controls-information-nda
- Subasri V, Krishnan A, Kore A, et al. “Detecting and Remediating Harmful Data Shifts for the Responsible Deployment of Clinical AI Models.” JAMA Network Open, 2025. https://pmc.ncbi.nlm.nih.gov/articles/PMC12138723/
- Feng J, Phillips RV, Malenica I, et al. “Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare.” npj Digital Medicine, 2022. https://www.nature.com/articles/s41746-022-00611-y
- Kapoor S, Narayanan A. “Leakage and the Reproducibility Crisis in ML-based Science.” 2022. https://arxiv.org/abs/2207.07048
- Kapoor S, Cantrell E, Peng K, et al. “REFORMS: Reporting Standards for Machine Learning Based Science.” 2023. https://arxiv.org/abs/2308.07832
- National Institute of Standards and Technology. “Artificial Intelligence Risk Management Framework (AI RMF 1.0),” NIST AI 100-1. January 2023. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf
- European Medicines Agency and Heads of Medicines Agencies. “Summary notes: meeting of the HMA/EMA group focused on AI with industry stakeholders, February 2026.” ema.europa.eu HMA/EMA AI group summary notes, February 2026 (PDF)
- Sáez C, et al. “Unsupervised Characterization of Temporal Dataset Shifts as an Early Indicator of AI Performance Variations: Evaluation Study Using the MIMIC-IV Dataset.” https://pmc.ncbi.nlm.nih.gov/articles/PMC12712564/
- Rahmani K, et al. “Assessing the effects of data drift on the performance of machine learning models used in clinical sepsis prediction.” medRxiv preprint, 2022. https://www.medrxiv.org/content/10.1101/2022.06.06.22276062v1








Your perspective matters—join the conversation.