Annex 22 Post-Implementation Audit: The First 90 Days After Go-Live

Why the First 90 Days Are the Real Test

Validation proves a model works under test conditions with test data on a defined day. Operations prove a model keeps working under real conditions with real data over time. The 90-day window is where those two worlds collide. It is long enough to catch drift, to accumulate a meaningful number of human-in-the-loop overrides, to see the first change requests that touch training data or thresholds, and to test whether the operational procedures written during validation are actually being followed on the floor. It is short enough that memory of the go-live decisions is still fresh and course correction is still cheap.

Annex 22 makes this point implicitly rather than explicitly. The draft language emphasises continuous oversight, model performance monitoring, and procedures for human review when necessary⁴. Nothing in the text prescribes a 90-day cadence. What the text does prescribe is that the operational reality of the AI system must be demonstrably controlled, and 90 days is the shortest practical horizon for demonstrating that control against multiple production batches, multiple deviations, multiple shift handovers, and multiple operator decisions.

The 9% figure is instructive. It tells us that even in AI-adjacent regulated industries, the discipline of post-deployment monitoring is the weakest link. Annex 22 is one of the first regulatory frameworks that treats post-deployment monitoring as a first-class deliverable rather than a nice-to-have, and organisations that have not built the muscle for it will feel the tension in the first 90 days.

What changes on day one that validation cannot anticipate

Real production data is never the same as the validation dataset. The distribution of inputs shifts because a supplier changes a raw material lot, because a season changes ambient humidity, because a batch record template gets a small revision that changes how a field is populated. Operators use the system in ways that the intended-use statement did not fully anticipate. A quality analyst might query the model for a batch that was flagged for an unrelated reason, or an engineer might run the model on historical data to test a hypothesis. None of these are misuse. All of them are edge cases the validation package did not enumerate.

The result is that the first 90 days generate an artefact stream the organisation did not previously produce: model outputs paired with human decisions on those outputs, plus commentary about why the human agreed or disagreed. This stream is the evidence base an inspector will eventually ask to see, and it is the evidence base internal audit needs to review before an inspector shows up.

The audit conversation shifts from design to evidence

Pre-Annex 22 audits of computerised systems were largely design conversations. Was the requirements specification complete. Was the risk assessment rigorous. Did the test protocols cover the requirements. Did the change control process function as documented. Those questions still matter, but the Annex 22 audit conversation adds a distinctly evidentiary layer. Did the model actually perform in production the way the validation predicted. Did the humans actually exercise the oversight the validation assumed. Did the change control process actually catch the changes that mattered. The shift is subtle but consequential: an inspector can no longer be satisfied by a well-organised validation binder. The inspector wants operational evidence, and the 90-day audit is where the organisation first tests whether that evidence exists in a form that can be shown.

The clock starts on multiple internal deadlines simultaneously

The first 90 days are also when several internal clocks begin ticking in parallel. The first periodic review clock. The first CAPA effectiveness verification clock for any deviations opened during hypercare. The first quarterly review of drift metrics. The first cycle of consumer complaints or downstream quality events that might touch model-supported batches. Coordinating these clocks is not the audit function’s job, but making sure the coordination exists is squarely within scope. When any of these clocks are running independently, without a coordinating quality lens, the audit will find gaps that no single function is positioned to catch.

The Six Gaps That Show Up in Every Post-Implementation Audit

Across early Annex 22 implementations, six patterns recur in the first-cycle audit. None of them are novel to AI. All of them are amplified by AI because the number of decision points, the volume of records, and the traceability expectations are higher than in a conventional deterministic system.

Gap one: the drift monitoring plan exists but nobody is actually running it

The validation package includes a drift monitoring procedure with thresholds, alert routing, and a defined review cadence. On paper it is complete. In practice the alerts route to a distribution list that no one actively monitors, the weekly performance review is happening but attendance is inconsistent, and the trending chart that shows model output distributions is being generated but not reviewed. This is the single most common finding because drift monitoring sits in an operational gap between quality, IT, and the process owner. Each function assumes another function owns the day-to-day rhythm.

Gap two: human-in-the-loop overrides are happening but not being captured with reasoning

Annex 22 and the parallel guidance from the EMA reflection paper both emphasise that human oversight is not just a checkbox⁸. The value of a human-in-the-loop step is only realised when the human’s reasoning is captured in a way that can be reviewed. In practice, operators are overriding model recommendations, but the audit trail records only the fact of the override, not the reason. When quality reviews the batch, they cannot tell whether the override reflects a legitimate operational judgement, a systematic issue with the model, or an operator working around a control they find inconvenient.

Gap three: change requests that touch the model are being routed as routine IT changes

An AI model has three change surfaces that a conventional computerised system does not: the training data, the model artefact itself, and the threshold or acceptance criteria that convert model outputs into decisions. A routine IT change control workflow will handle a version upgrade or a configuration change well. It handles a threshold change less well, because a threshold change is functionally a decision-boundary change even if it looks like a configuration edit. Post-implementation audits regularly find threshold changes that were logged as configuration and never triggered a validation impact assessment.

Gap four: the intended-use statement no longer matches how the system is being used

This is the most quietly serious finding. The intended-use statement locks in a specific scope: what the model does, what data it uses, what conditions it operates under, and what decisions it supports. Within a quarter of go-live, the system is often being used for adjacent purposes. A model validated to flag out-of-trend results in real time gets used to retrospectively investigate batches. A model validated to support one product family gets used across a second family that shares some but not all input characteristics. Neither expansion is inherently wrong. Both require an intended-use amendment and a validation impact review. Both are almost never captured.

Gap five: the training data lineage cannot be reproduced on demand

Annex 22’s data governance expectations, reinforced by the parallel Chapter 4 and Annex 11 revisions⁹, require that the data used to train and validate the model be traceable back to source with the same rigour as any GxP record. In the first 90 days, the audit finding is rarely that lineage does not exist. The finding is that lineage exists in a data science notebook, a shared drive, an S3 bucket, and a MLOps platform, and reconstructing it takes days. When an inspector asks the question, days is not an acceptable answer.

Gap six: the periodic review calendar has not been established

Validation ended, the system went live, and the periodic review that Annex 22 anticipates has not yet been scheduled. The audit finding is administrative rather than technical, but it matters because the periodic review is the mechanism by which drift monitoring, override analysis, change history, and intended-use scope are pulled together into a single documented assessment. Without a scheduled and calendared periodic review, the operational evidence stream generated in the first 90 days never gets consolidated, and the next inspection question becomes very hard to answer.

Where Annex 22 Rubs Against Existing MHRA and EMA Inspection Expectations

Annex 22 does not replace existing GMP expectations. It layers on top of them. That layering creates specific friction points, and post-implementation audit is where those friction points become visible. Understanding them matters because they shape both the audit findings and the corrective actions.

Friction one: Annex 11 says one thing about audit trail scope, Annex 22 implies more

Annex 11 as revised in the 2025 draft frames audit trail requirements around GxP-relevant events on a computerised system¹⁰. Annex 22 implies a broader scope for AI systems: not just what happened, but what the model recommended, what the human decided, why the human decided it, and what training-data or threshold context surrounded the decision. Internal audit needs to make a judgement call on whether the existing audit trail meets the broader implied scope. In most first-cycle audits, it does not, and the CAPA has to define what additional context is captured going forward.

Friction two: MHRA inspection expectations are increasingly explicit about AI in inspection responses themselves

The MHRA Inspectorate published guidance in June 2026 on the use of AI to draft inspection responses¹¹. The signal is that inspectors are becoming AI-literate faster than most quality organisations expect. A 90-day audit that produces a finding on an AI system will be read by an inspector who has seen many similar findings across many companies. Boilerplate CAPA language does not survive that scrutiny. Findings need to be closed with specific, verifiable, and testable actions.

Friction three: the static-model constraint creates operational rigidity

Annex 22 restricts AI in GMP-critical applications to static, deterministic models that do not update themselves in production¹². The constraint is defensible from a regulatory standpoint. Operationally it means every genuine performance issue triggers a controlled retraining event, and every retraining event is a formal change under change control. In the first 90 days, teams often discover that they underestimated how often a retraining cycle would be needed, and the change control queue backs up. The audit finding is not that retraining happened. The finding is that the volume of retraining events exceeded the capacity of the change control workflow.

Friction four: existing quality management systems are not structured to receive AI-specific evidence

Deviation systems, CAPA systems, and quality review boards are structured around conventional GMP artefacts. AI-specific evidence, particularly override logs, model performance trends, and drift alerts, does not fit neatly into those structures. Post-implementation audits regularly find that the AI evidence is being maintained but not integrated into the existing QMS, which means the evidence is not being surfaced to the quality decisions that most need it. The friction is organisational, not technical.

A Sample 90-Day Audit Checklist for AI in GMP

A workable 90-day audit does not attempt to replicate validation. It samples the operational evidence stream against the intended-use statement, the validation acceptance criteria, and the operational procedures written for live use. The following checklist is designed to be completed by a two-person audit team over roughly two weeks, with about half the time spent on evidence gathering and half on synthesis.

The Corrective Actions That Actually Work

Findings are only useful if the corrective actions that close them address the underlying condition rather than the surface symptom. Post-implementation audit CAPA design is where organisations either build durable AI governance or accumulate paperwork.

For drift monitoring gaps: assign a named owner and shift the review into an existing rhythm

The failure mode is almost always ownership ambiguity between quality, IT, and the process owner. The corrective action that survives is naming a single accountable owner and moving the drift review into an existing meeting rhythm rather than creating a new one. If the site quality council already meets weekly, the AI drift review is a standing agenda item on that council, with the named owner presenting. New meetings decay. Existing meetings persist.

For override capture gaps: change the override interface, not the training

Telling operators to write more thorough override reasoning does not survive shift pressure. What survives is making reasoning capture unavoidable at the interface level, with structured prompts for the most common override categories and a free-text field that is required rather than optional. The audit finding closes when a subsequent sample of overrides shows reasoning captured at an acceptable level of specificity.

For change control routing gaps: add a screening question to the change intake

The IT change intake form needs a screening question that flags any change touching an AI system for a validation impact triage. The triage does not need to be onerous. It needs to route threshold, model, and training data changes to a validation reviewer before the change is scheduled. Screening at intake is far cheaper than remediation after the fact.

For intended-use scope gaps: publish the scope statement in the user interface

The intended-use statement lives in a validation document that most users never see. Making the scope visible at the point of use, either as an in-application banner or a required acknowledgement on the first use each shift, dramatically reduces scope creep. The user does not need to read the full statement. They need to see, in context, what the system is for.

For training data lineage gaps: invest in a single documented lineage record

The corrective action is not building new tooling. The corrective action is producing a single documented lineage record that traces training and validation data from source to model input, stored alongside the validation package. If reproduction from source takes more than a day, a documented lineage record that can be produced in an hour is the interim answer. Tooling investment can come later.

For periodic review gaps: calendar the next 12 months

The corrective action is to calendar the next four quarterly reviews with named attendees and an agreed agenda template. Calendaring is administrative and immediate. It removes the ambiguity about whether reviews will happen and creates a forcing function for the evidence stream to be maintained.

For deviation and CAPA integration gaps: expand the root cause taxonomy

Existing deviation systems typically use a fixed taxonomy for root cause categorisation. That taxonomy rarely distinguishes between model performance issues, data quality issues at input, threshold or acceptance criterion issues, human-machine interaction issues, and infrastructure issues. Expanding the taxonomy to include those categories, even at a simple level, forces investigators to decompose AI-related deviations rather than defaulting to a generic root cause. The audit finding closes when a sample of subsequent AI-related deviations show root causes distributed across the expanded taxonomy rather than clustered under a single generic bucket.

For operator competency gaps: build decision-support scenarios into refresher training

Initial training tends to focus on system operation. Refresher training after the first 90 days is where organisations can introduce scenario-based decision support: here is a model recommendation, here is the surrounding context, what would you do and why. Running scenarios during shift huddles, with the outputs feeding back into the override reasoning capture, reinforces the judgement the system requires without adding a new training event. Operators develop the discrimination the system needs, and the organisation builds a record of that discrimination.

Escalation Triggers: When a Finding Becomes a Regulatory Risk

Not every 90-day finding rises to the level of regulatory notification. Most are internal issues to be closed through the normal CAPA process. A subset carries enough weight that quality leadership needs to make an explicit judgement about escalation. The following triggers should be built into the audit protocol so that the judgement is deliberate rather than accidental.

Turning 90-Day Findings into a Next-Cycle Improvement Plan

The temptation after a first-cycle audit is to treat the findings as a punch list, close them, and move on. That approach closes findings without changing the operational conditions that produced them, which almost guarantees the same findings return in the next cycle. A better approach treats the 90-day findings as diagnostic data about the maturity of the AI governance system, and uses that data to shape a multi-cycle improvement plan.

Segment findings by durability, not severity

The standard triage segments findings by severity. That is the right lens for prioritisation but the wrong lens for improvement planning. For planning, the useful segmentation is by durability of the underlying condition. A finding that can be closed by a single procedural change is a shallow finding. A finding that requires a workflow redesign, a system change, or a governance structure change is a deep finding. Shallow findings should be closed in the current cycle. Deep findings should be scheduled into a durable improvement roadmap, with milestones that align with the next scheduled internal audit and the next expected inspection.

Build a maturity view that survives leadership turnover

A maturity view articulates where the AI governance capability sits today and where it should sit in 12 months, expressed in terms of specific practices rather than abstract capability levels. What does drift monitoring look like at level one versus level three? What does human-in-the-loop reasoning capture look like at each level? The maturity view becomes the reference document for improvement planning and for onboarding new leaders. It is the artefact that keeps improvement momentum through leadership transitions, which are otherwise the point at which durable improvement programmes tend to lose oxygen.

Fold audit findings into the next validation cycle rather than treating them separately

The retraining events that Annex 22 anticipates create natural opportunities to fold audit findings into structured validation activity. A finding about training data lineage becomes a validation deliverable in the next retraining. A finding about threshold change control becomes a procedural update rolled out with the next model version. Bundling findings into validation activity is more efficient than running parallel CAPA and validation tracks, and it produces a validation record that reflects the operational learning of the previous cycle rather than repeating the original validation posture.

Track leading indicators, not just closure metrics

The metric most organisations track after an audit is CAPA closure rate. That metric is necessary but insufficient. The leading indicators worth tracking are the ones that would predict a repeat finding: override rate versus baseline, drift alert response time, time-to-reproduce training data lineage, percentage of changes that trigger validation impact triage. If those leading indicators improve, the next-cycle audit will produce fewer and shallower findings. If they do not improve, closure metrics will look good and findings will still recur.

Use the next-cycle plan to sequence the response to regulatory evolution

Annex 22 is not the endpoint. The June 2026 EMA workshop signalled that the regulatory position on generative AI and larger models in non-critical GMP applications may evolve¹³. The finalisation of Annex 11 and Chapter 4 in 2026 will introduce parallel expectations on data governance. The next-cycle improvement plan is the mechanism by which an organisation absorbs those changes without a series of standing-reset moments. Building the plan around durable practices rather than point compliance means the practices remain valid as the regulatory framework matures.

Conclusion

The first 90 days after Annex 22 go-live are not a validation extension. They are a distinct operational phase in which the assumptions of the validation package meet the ambiguity of live operation. Internal audit is the function best positioned to translate that meeting into structured evidence, and the audit findings that emerge are more valuable as a diagnostic than as a punch list. Organisations that treat those findings as data about the maturity of their AI governance capability, rather than as work to be closed and forgotten, will find themselves in a materially stronger position when the 2027 to 2028 enforcement phase begins and when the next generation of AI applications, potentially including regulated GenAI, becomes part of the operational landscape.

Sakara Digital works with pharma and biotech organisations building the operational discipline that Annex 22 will demand: audit programmes that produce durable improvement rather than checklist closure, quality management systems that can receive AI-specific evidence, and governance rhythms that survive leadership transitions. If your team is heading into a first-cycle audit or planning the next one and wants an independent perspective on where the operational risk actually sits, we are happy to have that conversation.