AI Incident Management in GxP: Classifying, Investigating, and CAPA-ing a Model Failure

Why Your Deviation System Has No Box for “The Model Was Wrong”

Deviation systems in pharma were designed around a specific mental model of failure. Something was supposed to happen a certain way, it happened a different way, and there is a discoverable reason: an operator missed a step, an instrument drifted out of calibration, a supplier changed a material, a procedure was ambiguous. The categories in your QMS reflect that history. Human error. Equipment failure. Material issue. Procedural gap. Facility or utility event.

A model failure does not fit any of those cleanly. The system did exactly what it was built to do. It took an input, applied learned parameters, and returned an output. Nothing broke. No alarm fired. The output was simply wrong, or was right in a way nobody could confirm, or was right and the human who reviewed it decided otherwise. Categorizing that as “equipment failure” is inaccurate and makes trending meaningless. Categorizing it as “human error” pushes the fault onto the reviewer and prevents anyone from finding the real problem.

So in practice, most of these events do not enter the QMS at all. They enter a Jira board or a ServiceNow queue. A model performance issue gets triaged by the team that owns the model. The team fixes it. The fix is real and often good. But there is no quality record, no impact assessment, no product or study linkage, no CAPA, no effectiveness check, and no trend. When the annual product quality review comes around, the AI system contributes nothing to it. When an inspector asks for the list of AI-related deviations in the last twenty-four months, the honest answer is that there is not one, and the reason is not that nothing went wrong.

What the regulators have already said

The regulatory position is not ambiguous, even where the specific guidance is still in draft. The EU’s draft GMP Annex 22 on artificial intelligence, released for consultation in July 2025, requires that models used in critical GMP applications have defined test metrics and acceptance criteria, are put under change control and configuration control before deployment, have their performance regularly monitored to detect changes, and have the input data monitored for drift against the documented sample space.¹ None of that is meaningful unless a failure of any of those controls produces a record and an investigation. Monitoring that never triggers a deviation is decoration.

The EMA’s reflection paper on the use of AI across the medicinal product lifecycle takes the same line from the other direction: a risk-based, human-centered approach across all phases, with sources of data and processing activity documented in enough detail to support traceability consistent with GxP expectations.² Traceability is only useful in retrospect. It exists so that when something goes wrong, you can walk backward from the wrong output to the reason.

And FDA’s long-standing expectation for investigations is not softened because software was involved. Under 21 CFR 211.192, any unexplained discrepancy or batch failure must be thoroughly investigated, the investigation must extend to other batches and other products that may have been associated with the failure, and a written record with conclusions and follow-up is required.³ A model that has been silently misclassifying inputs across multiple products is exactly the kind of shared cause that regulation was written for.

A Failure Taxonomy a Quality Organization Can Actually Use

A taxonomy is only useful if a QA reviewer with no machine learning background can read the category name, read the description, and recognize the event in front of them. Taxonomies written by data scientists tend to fail this test. Below is a seven-mode set that has held up in practice, with the description a quality reviewer needs and the detection method that makes the category real rather than theoretical.

The detection method matters more than the category name. A failure mode you have named but cannot detect is a failure mode you will only find when a customer, a regulator, or a patient finds it for you.

Why these seven and not more

Three of these modes are about the model’s relationship to its data. Drift, upstream data change, and out-of-distribution input are all versions of the same underlying condition: the input the model sees at run time no longer matches the input it was tested against. Drift monitoring literature separates data drift, where the input distribution changes, from concept drift, where the relationship between inputs and the outcome changes, and both are distinct from a straightforward upstream schema break.⁴ Keeping them separate in your taxonomy matters because they have different detection methods and different corrective actions. Drift is gradual and needs statistical monitoring. An upstream change is abrupt and needs a control on the change process. An out-of-distribution input is a single-record condition and needs a run-time gate.

Two modes are specific to generative systems. Fabricated output is a well-known behavior of language models and is the reason the draft Annex 22 excludes generative AI and large language models from critical GMP applications entirely, allowing them in non-critical uses only with a qualified human accountable for every output.¹ Prompt injection is the top-ranked risk in the OWASP Top 10 for LLM Applications, and the reason it is hard to eliminate is structural: models process instructions and data through the same channel, so content the system was asked to read can become content the system decides to obey.⁵

The last two modes are the ones most organizations leave out, and they are the ones that make the taxonomy honest.

Automation bias is the mode that quality organizations resist including, because it looks like it belongs in a training record rather than an AI incident. It belongs here. A controlled study of twenty-eight trained pathology specialists found that adding AI decision support improved overall performance but produced a measurable automation bias rate of seven percent, meaning cases where an initially correct human judgment was overturned by wrong AI advice. Time pressure did not change how often the effect occurred, but it made the effect worse when it did.⁶ A broader review of how medical professionals interact with AI-enabled decision support reaches similar conclusions about over-reliance and under-reliance both being real, measurable failure conditions.⁷

The draft Annex 22 is explicit that where a model provides input to a human decision, and where the testing effort was reduced because a human is in the loop, the training and consistent performance of that operator should be monitored the same way any other manual process is monitored.¹ That is a direct instruction to treat human deference as an in-scope failure mode with its own monitoring.

Triage: Getting an AI Failure Into the QMS Correctly

Once the taxonomy exists, the next problem is routing. Someone notices something. Who decides whether it becomes a deviation, and on what basis? The default answers are both wrong. “Everything becomes a deviation” buries the quality organization in low-value records and trains people to stop reporting. “The model team decides” means nothing becomes a deviation.

The workable rule is a two-question gate applied within one business day of detection, by the process owner and a quality representative together.

Near-misses are worth more than incidents

The AI incident reporting field has converged on a distinction worth borrowing. The AI Incident Database frames it by analogy to aviation, where an accident involves substantial damage or loss, and an incident is a case where the risk of an accident substantially increased.⁹ Research on standardizing AI incident reporting argues explicitly that near-misses should sit inside the reporting scope, because they carry nearly all the same diagnostic information as incidents and differ only in the outcome.¹⁰ Work from CSET on what a mandatory reporting regime would need makes a related point about the value of capturing events before harm materializes.¹¹

For a pharma quality system this is easy to implement because the concept already exists. You already have a near-miss or potential-deviation category. Point the AI failures at it. The result is that a model whose confidence-threshold gate rejected forty inputs last month produces a trend, and that trend is the earliest warning you will get of out-of-distribution conditions building up before something slips through.

Reproducibility: The Assumption Root Cause Analysis Makes and AI Breaks

Here is the part of AI incident management that quality organizations are least prepared for, and it is not a tooling problem. It is an assumption problem.

Every root cause method in the pharma toolkit assumes reproducibility. Five whys assumes you can establish what happened. Fishbone assumes you can test candidate causes against the event. Fault tree analysis assumes a deterministic chain. Laboratory investigations under a phase one and phase two structure are built around retesting: you go back to the sample, you rerun the method, and the result either repeats or it does not, and either answer tells you something. The whole apparatus rests on the ability to put the system back in the state it was in and observe the failure again.

With an AI system, you often cannot do that. There are three separate reasons, and they compound.

The nondeterminism is real and it is technical

It is worth understanding the mechanism, because engineering teams sometimes claim reproducibility is achievable by setting a random seed and it usually is not. Research on floating-point non-associativity has documented how run-to-run variability arises in both high-performance computing and deep learning workloads from rounding differences and nondeterministic behavior in GPU kernels, particularly in operations such as fused attention and normalization layers.¹² More recent work identifies a systematic cause in inference serving specifically: hidden batch-size dependence in GPU kernels, where the numerical result for your request depends on which other requests happened to be batched with it. That means the same input, on the same hardware, with the same weights and a temperature of zero, can still produce a different output depending on server load at the moment of the call.¹³

This is fixable at an engineering level, at some effort and some throughput penalty, and if you are running a model in a critical GMP application you should ask whether it has been fixed. But you should not assume it. And you should never build an investigation procedure whose first step is “reproduce the failure” without a defined path for what happens when that step fails.

Investigating When You Cannot Reproduce the Failure

The techniques below are what a competent investigation looks like when reproduction is unavailable. They are not substitutes for reproduction where reproduction is possible. They are the fallback path, and your deviation procedure should name them explicitly so that investigators are not improvising under time pressure.

1. Retrospective pinning

Before anything else, reconstruct the exact artifact set that was live at the time of the event. Model version and weight file hash. Serving image or library version. Prompt template version, if applicable. Retrieval index snapshot identifier. Feature pipeline commit. Configuration values including any threshold settings. Hardware or serving tier. Vendor model identifier and version string if the model is external.

If your model registry records all of these against a deployment date, this takes an hour. If it does not, this is where the investigation dies, and the finding to write is not about the model at all. It is that the system lacked the configuration control that draft Annex 22 requires before deployment.¹ That is a real, defensible root cause and a much better outcome than “not reproducible.”

2. Treat the inference log as the batch record

If you cannot rerun the model, the log of what it actually did is your only primary evidence. This changes what the log has to contain. A log that records only a request identifier and a latency is an operations log. A log that can support a GxP investigation records the input as received, the output as returned, the model and artifact versions in force, the confidence score, the identity of any human who reviewed the output, and what that human decided.

The draft Annex 22 already points at part of this: it expects systems using models to predict or classify to log the confidence score for each prediction or classification outcome, and to have a threshold setting that flags an outcome as undecided rather than returning an unreliable answer.¹ A confidence score you logged is an investigative asset. A confidence score the model computed and discarded is nothing.

3. Shadow replay

Where you cannot reproduce the single event, you can often replay a population. Take the historical inputs you retained, run them offline against the suspect version and against a known-good or corrected version, and compare outputs at scale. This does not tell you why one specific record went wrong. It tells you how often the suspect version disagrees with the reference, on which segments of the input space, and whether the disagreement rate changed over time. For a look-back scoping decision, that is more useful than reproducing one case.

Shadow replay only works if you kept the inputs. This is the single strongest argument for input retention, and it is the point at which retention policy stops being an IT decision and becomes a quality decision.

4. Reason from the input distribution, not the single case

This is the mental shift that matters most. When a model fails on record 4471, the instinct is to study record 4471. Often the more productive question is: what characteristics does record 4471 have, how many other records share them, and how did the model perform on that whole group?

Answering that turns an unreproducible single event into a measurable population statement. You may not be able to say why the model got that record wrong. You can very often say that records with a particular attribute combination represent three percent of throughput, were underrepresented in the test set, and show an error rate several times the validated rate. That is a root cause you can write, defend, and correct.

5. State the uncertainty explicitly

ICH Q9(R1) reframed quality risk management around the level of formality appropriate to the importance, uncertainty, and complexity of the decision, and it named high subjectivity in risk assessments as one of the specific problems the revision set out to reduce.¹⁴ An AI investigation is a place where uncertainty is genuinely high and pretending otherwise is the greater risk.

Write down what you established, what you bounded, and what you could not determine, and be specific about each. “We could not reproduce the failure because the retrieval index was reindexed on 14 March and the prior snapshot was not retained. We bounded impact by replaying 22,000 retained inputs against the current index and comparing to reviewer decisions.” That paragraph is defensible. “Root cause could not be determined” is not.

GxP Impact Assessment and the Look-Back Scoping Question

Impact assessment for an AI failure has two parts. The first is straightforward and most organizations do it adequately: what did this output touch? The second is where these investigations get expensive, and where most of them go wrong: how far back does the investigation have to reach?

Part one: what did the output touch

Trace the output forward through every downstream use. The categories that change the character of the investigation are:

• A batch record or release decision. The output became part of the manufacturing record or contributed to disposition. This puts 21 CFR 211.192 squarely in scope, including the requirement to extend the investigation to other batches and other products that may have been associated with the same failure.³
• A regulatory submission. Data or analysis produced or screened by the model reached a filing. This raises the question of whether a correction or notification to the agency is required, and it pulls regulatory affairs into the investigation from day one rather than at close-out.
• A safety report or signal assessment. The model was involved in case intake, coding, seriousness or expectedness assessment, literature screening, or signal detection. Missed cases and miscoded cases have their own reporting obligations that run on their own clocks.
• A clinical data set or subject-level decision. The output affected eligibility, randomization, data review, or endpoint adjudication in a study.
• A GxP record with retention obligations. Even where no product decision was affected, if the output is part of a retained record, the record’s accuracy is in question and the correction has to be traceable.

Part two: the look-back scope, which is the expensive question

For an abrupt failure, look-back scoping is easy. An upstream feed changed on a known date, so the exposure window runs from that date to detection. Done.

For drift, there is no date. The model did not fail on Tuesday. It got slightly worse every week for seven months, crossed the acceptance threshold at some unknown point, and was noticed when someone happened to compare a batch of outputs against confirmed results. The question “how far back do we go” has no obvious answer, and the range of defensible answers spans from six weeks to two years. That range is the difference between a two-week investigation and a program of work with a real budget attached.

Use four anchors, and take the earliest one you cannot rule out.

Making the look-back affordable

An unbounded look-back on a high-volume model is not practical to do exhaustively, and it does not have to be. Stratify and sample. Use the retained confidence scores to rank records by the likelihood the model was uncertain. Run shadow replay against a corrected version and focus human review on the disagreements. Use subgroup analysis to identify which segments of the input space carry the elevated error rate, then bound exposure to those segments with a documented rationale for why others are unaffected. Escalate to full census review only for the segments and time periods where the sampled error rate is material.

The rationale for that sampling plan is itself a quality document, and it should be approved before the review starts, not written up afterward to justify what was done.

When the incident leaves the QMS

Some AI failures carry external reporting obligations that run on shorter clocks than a typical deviation investigation. Under Article 73 of the EU AI Act, providers of high-risk AI systems must report serious incidents to the relevant market surveillance authority, with an outer deadline of fifteen days from awareness, reduced to two days where the incident involves a widespread infringement or serious harm to health or safety, and ten days where a death has occurred, with immediate reporting expected once a causal link is suspected. The authority must respond within seven days.⁸ The European Commission has since consulted on draft guidance and a reporting template that fill in the operational detail.¹⁵¹⁶

The practical consequence for a life sciences quality organization is that a serious AI incident may need to be reported before the investigation has established root cause. Your deviation procedure needs a trigger that fires at intake, not at close-out. That is not a new problem: field alert reports and expedited safety reporting already work this way. It is a matter of adding the AI Act trigger to the same intake screen.

CAPA That Is Not “Retrain the Model”

Nearly every AI deviation you will read has the same corrective action: retrain the model on updated data. Sometimes with a second line: enhance monitoring.

Retraining is a correction. It is the equivalent of reworking the affected batch. It restores the system to an acceptable state and it addresses nothing about why the system reached an unacceptable state without anyone noticing. A CAPA whose corrective action is retraining is a CAPA that guarantees the same event recurs, because every condition that allowed it is still in place.

A real CAPA for an AI failure addresses three specific gaps. All three are almost always present, and naming them is what makes the CAPA credible.

Root cause statements to reject at review

The CAPA review board should send back any AI deviation whose root cause is one of the following, because none of them is a cause:

• “Model limitation.” This restates the observation. Every model has limitations. The question is why this limitation was not identified during qualification, or was identified and not controlled.
• “AI is probabilistic.” True and irrelevant. If the probabilistic behavior was known, an acceptance criterion should have bounded it and a monitor should have watched it. If it was not known, that is a qualification gap.
• “User error” where a human accepted a wrong model output. This is automation bias, which is a designed-in risk of human-in-the-loop controls and a known, studied effect.⁶ The cause is that the review control was designed assuming an attentive, independent reviewer without measuring whether reviewers behave that way.
• “Insufficient training data.” Closer, but incomplete. The cause is that the input sample space was not fully characterized, or that a subgroup was known to be underrepresented and no run-time gate was placed on it.
• “Vendor issue.” A vendor changing a model behind an API is a real event, but it is your change control gap. The corrective action is contractual notification requirements and version pinning, not an entry blaming the supplier.

Effectiveness checks that actually test something

The hardest part of an AI CAPA is proving the corrective action worked, because the desired outcome is the absence of an event. Waiting ninety days and observing no recurrence proves very little when the original failure took seven months to surface.

Three effectiveness checks work better:

• Seed a known condition and measure detection latency. Deliberately introduce a controlled drift or a set of known-bad records into the monitored stream and measure how long the new control takes to flag it. This tests the control, not the absence of failure. Document it as a challenge test with predefined acceptance criteria.
• Run the challenge set on a schedule and trend the result. A fixed set of records with verified correct answers, run monthly, gives you a performance time series that does not depend on waiting for confirmed real-world outcomes.
• Verify the change control linkage by test. Make a controlled change to an upstream source and confirm that the change control workflow surfaces the dependent model as an affected system. If it does not, the corrective action was documented but not implemented.

What Has to Exist Before the Incident

Everything above depends on evidence that has to be created before anyone knows an incident is coming. This is the section to act on if you read nothing else, because it is the only part that cannot be done retrospectively.

Inference logging with GxP-grade content

For every model used in or adjacent to a GxP process, decide what a single inference record must contain and enforce it. A working minimum:

Version pinning and a model registry that is a controlled record

Draft Annex 22 requires that a tested model be placed under change control and configuration control before deployment, with effective measures to detect unauthorized change, and that any change to the model, the system, or the process it operates in be documented and evaluated for whether retest is needed, with any decision not to retest fully justified.¹ The ISPE GAMP Guide on artificial intelligence, published in July 2025 to sit alongside GAMP 5 Second Edition, addresses the same lifecycle expectations for AI-enabled systems in GxP processes.¹⁷

In operational terms that means a registry entry per deployed model that records the artifact hashes, the intended use description, the input sample space, the acceptance criteria and the test results against them, the deployment date, the change history, and the owner. When an incident occurs, that entry is the first document the investigator opens. If it does not exist, the investigation starts three days behind and never catches up.

Retention set by the record, not by the log platform

Observability platforms default to short retention because storage has a price and operations teams rarely need last year’s traces. GxP records do not work that way. If a model contributed to a batch record, the evidence of what the model did is part of the reconstruction of that batch, and its retention should track the retention of the record it supported, not the default of the logging tool.

This is a decision quality has to make and fund, and it should be made deliberately, per model, based on the criticality of what the model touches. Storing full input payloads for a high-volume vision model is genuinely expensive. Storing hashes, feature summaries, confidence scores, and a sampled subset of full payloads is much cheaper and preserves most of the investigative capability. Make that trade-off explicitly, write down the rationale, and revisit it at periodic review.

Preserved reference data and challenge sets

Draft Annex 22 requires that test data independence be maintained by technical and procedural controls, that test data be protected by access control and audit trail, and that there be a record of which data was used for testing, when, and how many times.¹ Keep that discipline through the operational life of the model, not only through qualification. The qualification test set is the reference point against which you will measure whether performance has moved. If it has been contaminated by being used for retraining, you have lost your only fixed measuring point and every subsequent performance claim is contestable.

Separately, build a challenge set: a small, fixed set of records covering the known-hard cases, the rare subgroups, and the boundary conditions, with verified correct answers. Run it on a schedule. It is the cheapest silent-failure detector available and it produces a performance trend that does not depend on waiting for real-world outcomes to be confirmed.

Where this connects to governance you already have

None of this requires a new governance body. The NIST AI Risk Management Framework organizes AI risk work into govern, map, measure, and manage functions, with incident response, recovery, communication, and documented lessons learned sitting inside the manage function and feeding back into governance.¹⁸ Mapped onto a pharma quality system, that is a deviation and CAPA process feeding a management review. You have both. The work is connecting them to the AI estate, not building a parallel structure.

Practically, that means the AI model inventory becomes an input to the deviation intake screen so a reporter can select the model. It means the change control form gains a question about whether any registered model consumes the affected data or system. It means the periodic review of an AI system includes the deviation and near-miss trend for that model. And it means the CAPA review board has one person who can read a drift chart. That is a modest set of changes, and it is the difference between a quality system that can hold an AI failure and one that quietly drops it.

Conclusion

The instinct when AI enters a regulated process is to build something new for it: an AI incident procedure, an AI review board, an AI risk register. Those artifacts get written, approved, and then not used, because the people doing the work already have a deviation system and will use the system they know. The more durable approach is to make the existing quality system capable of holding an AI failure, which takes a taxonomy people can apply, an investigation path that does not assume reproducibility, an impact assessment method that can scope a look-back for gradual degradation, and a definition of corrective action that refuses to accept retraining as an answer.

The hardest of those is reproducibility, and it is hard in a way that is easy to miss until the first real investigation. Root cause analysis in pharma has always rested on the ability to go back and look again. With a probabilistic model, an unpinned artifact chain, or an input that no longer exists in its original form, that ability is gone, and the investigation has to move from establishing what happened in one case to establishing a rate across a population. Organizations that have built the logging, pinning, and retention to support that shift can investigate an AI failure. Organizations that have not will write “could not be reproduced” and hope the question does not come back.

Sakara Digital works with pharma and biotech organizations building AI into GxP processes and the quality systems that have to hold them accountable. If you are working through how model failures should enter your deviation and CAPA process, or want an independent read on whether your current logging and version control would survive a real investigation, we are happy to have that conversation.

For Further Reading