The AI Model Risk Assessment for Pharma: A Structured Checklist

Why Proportionate Risk, Not Uniform Risk

Every mature quality system in pharma already operates on proportionate risk. ICH Q9(R1), the international standard for quality risk management, is explicit that “the level of effort, formality and documentation of the quality risk management process should be commensurate with the level of risk”². Yet when AI arrives, teams frequently abandon that principle and default to one of two extremes: either they treat every model as an experimental science project outside the quality system, or they apply the full GxP validation package to everything, including tools that would never touch a patient or a batch record.

The regulatory signals point firmly toward proportionality. The FDA’s January 2025 draft guidance on AI in drug and biological product regulation introduces a seven-step credibility framework built on the concept of a “context of use,” where the required evidence scales with the model’s risk to the regulatory decision it informs³. The joint FDA-EMA Guiding Principles of Good AI Practice in Drug Development, released January 14, 2026, name “a risk-based approach” as one of ten foundational principles⁴. The EMA’s draft Annex 22 to EU GMP, published in July 2025, states directly that manufacturers should perform a risk assessment consistent with ICH Q9 for each AI use case and let that assessment inform how much validation and review are documented in the AI quality plan⁵.

Proportionality is not just about efficiency. It is about credibility. If your risk assessment produces the same output for a marketing copy generator and a pharmacovigilance signal detection model, inspectors and auditors will conclude that the assessment is not actually assessing anything. A proportionate framework, by contrast, produces a defensible narrative: this model earned that level of scrutiny for these documented reasons, and the reasoning can be traced from category scores to control decisions.

The Seven Risk Categories for Pharma AI

The categories below are the ones we have found most useful when adapting general AI risk management frameworks like NIST AI RMF and Deloitte’s Trustworthy AI Framework⁸ to pharma-specific contexts. They map to what regulators, quality units, security teams, and business owners each need to see, and they align with the trust characteristics named in the joint FDA-EMA principles and the AI TRiSM market Gartner has been defining since 2023⁹.

Category 1: Patient Safety Impact

The most consequential category. This asks: if this model produces a wrong output, could a patient be harmed? Direct harm includes AI that informs dosing, adverse event triage, pharmacovigilance signal detection, or diagnostic support. Indirect harm includes AI that gates manufacturing releases, manages cold chain, or controls a critical process parameter. Even models that seem administrative can carry patient safety weight if they feed decisions that eventually touch a patient.

Category 2: Regulatory Exposure

This category asks whether the model sits inside a regulated activity, and if so, under which framework. A model that informs a drug submission, a GMP release decision, a GCP monitoring plan, or a GVP case triage will be examined against GxP validation expectations, the FDA’s credibility framework, EMA Annex 22 in the EU, and potentially the EU AI Act’s high-risk system requirements if it functions as a safety component of a medical device or in vitro diagnostic. Models used purely internally for exploratory research generally carry lower regulatory exposure, though intellectual property and confidentiality obligations still apply.

Category 3: Financial Materiality

What is the financial consequence if this model produces wrong or biased outputs? A pricing model, a forecasting model that drives production planning, a claims analytics model, or an AI that scores commercial opportunities can have direct revenue implications. A model that supports fraud detection in speaker programs or hub services can trigger settlement exposure. Financial materiality does not just mean the size of the bet; it also means how quickly a wrong output could accumulate loss before it is caught.

Category 4: Data Privacy

Does the model process personal data, and if so, of what kind? Models that touch patient-level real-world data, clinical trial data, HCP data with identifiable prescribing patterns, or employee data carry both GDPR and HIPAA weight. Even models that only see aggregated or anonymized inputs need this category assessed, because re-identification risk in genomic or rare-disease datasets is not zero and is often underestimated. Cross-border data flows amplify this category significantly.

Category 5: Security

What is the attack surface, and what would an adversary gain by manipulating this model? Generative AI systems accessible to external users create new prompt-injection and data-exfiltration risks that traditional application security programs do not fully address. Gartner has predicted that through 2026, at least 80% of unauthorized AI transactions will come from internal violations of policy rather than external attacks¹⁰, which means the security category needs to cover insider misuse as much as external breach. Models that pull sensitive documents into a retrieval pipeline are particularly exposed.

Category 6: Operational Continuity

What happens to the business if this model becomes unavailable, or if its outputs need to be paused? Some models are enrichments to processes that could revert to manual work; others sit on the critical path for high-volume decisions that cannot easily be done by humans in the required time window. This category is where third-party AI dependencies show up: if the model relies on a foundation-model API whose availability, terms of use, or output distribution can change, the operational risk lives here. Concentration risk on one provider is worth naming explicitly.

Category 7: Model Drift

How sensitive is the model to changes in its inputs, its operating environment, or the phenomenon it is trying to predict, and how would you detect a problem? A model trained on 2023 prescribing patterns will not remain accurate in 2027 without monitoring and retraining. A generative model whose underlying foundation model version changes will not produce identical outputs across versions. Industry guidance now treats explainability and drift detection as formal requirements for AI-enabled systems, not optional enhancements¹¹. The drift category is where you name what you will monitor, how often, and what the tripwires are.

Scoring Rubrics: Turning Judgement Into Numbers

A category is only useful if two independent assessors would score the same model similarly. The rubrics below give a 1-5 scale for each category, with anchor descriptions at each level. The point is not mathematical precision. It is repeatability: the ability to defend a score by pointing to a specific rubric level and explaining why the model matches it.

Patient Safety Impact Rubric

Regulatory Exposure Rubric

Financial Materiality Rubric

Data Privacy, Security, Operational Continuity, and Model Drift Rubrics

The same 1-5 structure applies to the remaining four categories. For data privacy, anchors run from “no personal data touched” through “aggregated data only” to “large-scale processing of sensitive-category data including patient health data.” For security, from “isolated internal use, no external inputs” to “internet-exposed generative AI with sensitive data in retrieval context.” For operational continuity, from “outage of any duration has no impact” to “outage of one hour creates unrecoverable business loss.” For model drift, from “static rules with no drift risk” to “high-dimensional foundation model with rapidly changing inputs and no drift monitoring in place.” The full anchor set is worth putting on a single laminated card that every model owner keeps at hand.

The AI Risk Register: What It Actually Contains

The AI risk register is a living document that consolidates every model your organization runs, along with its risk scores, controls, owners, and monitoring commitments. If your existing enterprise risk register is where board-level risks are tracked, the AI risk register is a specialized instance that feeds into it. Regulators, auditors, and inspectors increasingly expect one to exist and to be current¹².

At minimum, each row in the AI risk register should carry the following fields.

The composite risk tier is where scores collapse into a decision. Different organizations will weight categories differently, but a defensible starting rule is: any category score of 5 sets the model at Tier 1; any two category scores of 4 or higher set it at Tier 2; average category score of 3.0 or higher sets it at Tier 3; everything else is Tier 4. Whatever formula you choose, document it and apply it consistently. Do not let the tier be argued case by case, or the register loses its integrity.

Integrating With ERM and ICH Q9 Quality Risk Management

An AI risk assessment that lives outside your existing enterprise risk management (ERM) and quality risk management (QRM) programs will be brittle. It will duplicate work, create parallel governance conversations, and eventually be quietly deprioritized when leadership attention moves on. The organizations that make this stick treat the AI risk assessment as a specialized input to the risk systems that already exist, not as a competing program.

The ERM Integration

ERM is where your organization tracks risks that could affect strategic objectives, financial performance, reputation, and regulatory standing at the enterprise level. The AI risk register should feed into ERM via a documented process: any model at Tier 1 or Tier 2 should have a corresponding entry in the ERM register with the same risk owner and control statements. When ERM reports go to the audit committee, the AI risk register should be a named data source, not a separate presentation. This aligns with the emerging view that ERM is quietly becoming pharma’s AI control tower, linking GRC, validation, and regulatory readiness¹³.

The QRM Integration

QRM under ICH Q9(R1) applies to any activity that could affect product quality, patient safety, or data integrity. Every AI model with a patient safety score of 3 or higher, or a regulatory exposure score of 3 or higher, needs to be visible in the QRM system. Practically, that means the same category scores flow into the site or global QRM committee’s review cadence, and any change to the model that would materially affect the scores triggers a QRM review, not just an IT change record. This is how Annex 22’s requirement to perform a risk assessment consistent with ICH Q9 for each AI use case is operationalized⁵.

The AI Management System Integration

Organizations pursuing ISO/IEC 42001 certification for their AI management system will find that the risk assessment described here maps neatly to the standard’s requirements. ISO/IEC 42001 explicitly requires organizations to conduct AI impact assessments, maintain risk registers, and implement risk-based controls proportionate to the AI system’s impact¹⁴. Structuring the risk assessment this way from the start means the ISO certification effort becomes a documentation exercise, not a rebuild.

The Full Assessment Checklist

The checklist below is the working document a model owner completes for every AI system before it moves out of development, and refreshes on a cadence tied to its risk tier. It is deliberately short. Long checklists are ignored; short ones with real teeth get completed.

The Escalation Matrix: Who Decides at Each Risk Level

An escalation matrix names, in advance, who has decision authority at each risk tier. This is the part of the framework that people most want to skip and most regret skipping. When a Tier 1 model reaches launch readiness, no one wants to be the person deciding on the fly who signs it off. The matrix removes that friction and, more importantly, creates a defensible record that the right level of oversight was applied.

Two design principles are worth calling out. First, approval authority at Tier 1 must include roles with independent standing, meaning people who can say no without career risk. If the same executive owns both the delivery and the approval, the approval is not real. Second, the escalation matrix should include a route for whistleblowing or bypass concerns. Someone lower in the organization must be able to escalate a concern about a Tier 3 or Tier 4 model that they believe is misclassified, without going through the model owner. A quiet channel to the head of quality or the chief compliance officer works well in most organizations.

Common Mistakes and How to Avoid Them

Mistake 1: Treating All AI Models With Equal Risk

This is the mistake this article is really about. It shows up in two shapes. The first is over-controlling low-risk models: requiring the same validation package for a marketing copy generator as for a signal detection model, then quietly ignoring the process because it is impossibly expensive to apply at scale. The second is under-controlling high-risk models: waving them through because the organization has decided AI is a strategic priority and slowing down is politically difficult. Both are eventual audit findings. The proportionate framework in this article exists to make the difference between the two visible and defensible.

Mistake 2: Confusing the Model Registry With the Risk Register

A model registry is a catalog of models, their versions, their owners, and their technical characteristics. A risk register is a catalog of risks, their scores, their controls, and their governance decisions. They intersect (every model in production should appear in both), but they are different artifacts with different audiences. The registry answers “what do we have running.” The risk register answers “what could go wrong and what are we doing about it.” Merge them at your peril; you will end up with a document that satisfies neither engineering nor governance.

Mistake 3: Static Assessments

Industry guidance now treats AI validation as a lifecycle activity, not a one-time event¹⁵. Yet many organizations still run the risk assessment once, at launch, and never touch it again. The world moves. Foundation model versions change. Upstream data sources drift. The regulatory landscape evolves; a model that was Tier 3 in early 2026 may be Tier 2 after the EU AI Act’s high-risk provisions come into full force in August. The review cadence is not optional. If the cadence cannot be met with available governance capacity, that is a signal to either invest in capacity or narrow the AI portfolio, not to skip the reviews.

Mistake 4: Making the Assessment Too Long

The best AI risk assessment in the world is worthless if model owners avoid completing it. We have seen 40-page assessment templates that require six weeks to complete. Predictably, they are only filled out for the two or three models a year that leadership specifically pushes. Everything else runs unassessed. A short, high-signal assessment (the checklist in this article fits on two pages) gets completed. A long, comprehensive one becomes a bottleneck people route around.

Mistake 5: Confusing Validation With Governance

Validation asks whether the model works as intended. Governance asks whether the organization should be using this model at all, and how it will be overseen if the answer is yes. Both are necessary. Neither substitutes for the other. A model can be technically validated and still be governed poorly (no clear owner, no monitoring, no escalation path). A model can be governed well and still fail validation (poor training data, unrealistic performance claims). Keep the two conversations distinct, and make sure both happen for every Tier 1 and Tier 2 model.

Mistake 6: Underestimating Third-Party and Foundation Model Risk

A material share of the AI running in pharma today is not built in-house. It is consumed via APIs from foundation model providers, embedded in SaaS applications, or delivered by specialized vendors. Yet risk assessments frequently treat these as if the risk stopped at the boundary of the enterprise firewall. It does not. When the foundation model behind your regulatory-document drafting tool is updated, the outputs your teams get change. When your vendor’s terms of service change to permit use of your prompts for model training, your data governance posture changes. When a critical AI-powered pharmacovigilance vendor experiences an outage during a signal review deadline, your operational continuity is affected regardless of who owns the model. The risk assessment needs to explicitly address third-party dependencies: which vendor, which model version, which contractual protections, which exit strategy if things go wrong. If this is not visible in the risk register, it is not being managed.

A Worked Example: Two Models, Two Very Different Tiers

Frameworks come alive when applied to concrete cases. Consider two AI models a mid-sized biopharma might deploy in the same quarter.

Model A is a generative AI assistant that helps commercial teams draft first-pass talking points for advisory board meetings. It reads publicly available scientific literature and internal, non-patient-level meeting summaries. Human reviewers always edit the output before use. It scores 1 on patient safety (no clinical decision path), 2 on regulatory exposure (commercial use only, no promotional claims generated), 2 on financial materiality (small productivity gain, no revenue exposure), 2 on data privacy (no personal data), 3 on security (external foundation model, sensitive commercial context), 2 on operational continuity (revert to manual drafting), and 3 on model drift (foundation model versioning). Average score: 2.1. Tier 4. The assessment takes a model owner about 45 minutes to complete, is self-attested, refreshed annually, and satisfies the governance requirement without slowing anything down.

Model B is a machine learning model that supports pharmacovigilance signal triage by scoring incoming case reports for probability of being a validated safety signal. It runs against real patient-level adverse event data. Safety scientists review every high-scored case, but the model’s ranking affects which cases get attention first. It scores 4 on patient safety (indirect impact via triage prioritization, human review present), 5 on regulatory exposure (GVP-regulated, direct submission relevance), 4 on financial materiality (missed signal exposure is material), 5 on data privacy (patient health data at scale), 3 on security (internal system, sensitive data), 4 on operational continuity (backlog builds rapidly if unavailable), and 4 on model drift (real-world drug and disease patterns shift). Any score of 5 sets it to Tier 1. Full validation, quarterly executive AI Governance Board review, independent annual audit, documented human oversight design, KRIs for triage accuracy and drift, external regulatory affairs review. Very different treatment, driven by the same framework.

The value of the framework is not that it produced these specific tier assignments. It is that two different assessors, working independently, would produce essentially the same tier for each model, and that the decision-makers at each level know in advance what they are being asked to sign off on. That is what a proportionate risk framework buys you: consistency, defensibility, and speed at the low end paired with genuine scrutiny at the high end.

Conclusion

The AI models running in pharma today are not going to be governed by a heroic case-by-case act of judgement. They are going to be governed by a small number of standardized artifacts (a scored risk assessment, a risk register entry, an escalation record) applied consistently across a portfolio that will only grow. The seven categories, the 1-5 rubrics, the four-tier structure, and the escalation matrix in this article are a starting point, not the last word. Adapt them to your organization’s risk appetite, regulatory footprint, and existing governance vocabulary. The specific choices matter less than the fact of having a defensible, repeatable, proportionate framework you actually use.

Sakara Digital works with pharma and biotech organizations building this kind of proportionate, defensible AI risk assessment structure from the ground up, or shoring up assessments that have grown ad hoc. If you are exploring how to integrate AI risk assessment with your existing ERM and quality risk management programs and want an independent perspective on where to start, we are happy to have that conversation.