Human-in-the-Loop Requirements for Pharma AI: What FDA and EMA Actually Expect

What Regulators Actually Mean by “Human Oversight”

Read enough regulator-issued AI documents and a pattern emerges. The FDA’s January 2025 draft guidance on AI to support regulatory decision-making for drug and biological products anchors its framework in a seven-step credibility assessment that culminates in a fitness-for-purpose judgment reached by human evaluators.¹ The EMA’s September 2024 reflection paper on AI in the medicinal product lifecycle repeats the phrase “human oversight” without ever fully operationalizing what it looks like at scale.² The EU AI Act, which unlike guidance is binding law, requires that high-risk AI systems be designed and developed so that they can be “effectively overseen by natural persons during the period in which they are in use.”³

Each of these documents assumes the reader knows what human oversight means in practice. None of them define it in a way that is directly executable in a validated GxP environment. That is not a criticism of the regulators; it is an accurate description of where the field is. The regulatory language is deliberately outcome-based, and the burden of translating it into workflows falls on sponsors, manufacturers, and their auditors.

The translation problem is what this article addresses. Vague language becomes usable only when a sponsor can answer four questions for a given AI use case: What decision or output requires oversight? At what point in the workflow does oversight occur? What does the reviewer actually do? And what evidence proves the oversight happened? The regulatory expectation is that all four have documented answers, that the answers scale with risk, and that the answers hold up under inspection.

The gap between policy language and implementation reality is not unique to AI. The same gap existed when computerized system validation was still being codified, when electronic records requirements first appeared, and when data integrity guidance moved from a series of memos into a formal ALCOA+ framework. In each case, the initial regulatory language sounded aspirational until enforcement made it concrete. Sponsors that treat the current AI-oversight language as similarly translatable, and start building the internal artifacts now, will not be caught by the coming waves of inspections. Sponsors that wait for a finding to define what “effective oversight” means will find that the definition is being written into their own inspection report.

The Three Oversight Models: In-the-Loop, On-the-Loop, In-Command

The literature and the regulators use three overlapping but distinct terms for human involvement with AI systems. The terms are not interchangeable, and choosing the wrong model for a given use case is where a lot of pharma AI governance goes wrong.

Human-in-the-Loop (HITL)

Human-in-the-loop describes governance where a human is inserted directly into the AI decision pathway and the AI cannot proceed to action without explicit human approval at defined checkpoints.⁴ The AI generates a recommendation; the human accepts, rejects, or modifies it; the accepted output becomes the record. HITL is the highest-friction model, is required where errors are consequential and reversibility is limited, and is the default expectation for AI outputs that directly influence patient safety, product release, or regulatory submissions.

Human-on-the-Loop (HOTL)

Human-on-the-loop describes governance where the AI executes autonomously within defined operational boundaries while a human monitors performance and can intervene when the system operates outside expected parameters.⁵ The distinguishing feature is that intervention is exception-driven, not routine. HOTL is appropriate for high-volume routine tasks where synchronous review would be a bottleneck, provided there are validated alarm thresholds, a defined intervention protocol, and evidence that the human monitor has the capacity and authority to actually stop the system when needed.

Human-in-Command

Human-in-command describes the highest, policy-level oversight where humans define the AI’s purpose, boundaries, and lifecycle rules but do not necessarily see individual outputs.⁶ This is the oversight model that governs an AI program as a whole: the decision to deploy the AI, the risk classification, the validation protocol, the retirement criteria. Human-in-command sits above HITL and HOTL and is the mode most closely aligned with the EMA’s framing of “expert judgment” and the EU AI Act’s requirement that natural persons be assigned oversight responsibility.

A well-run pharmaceutical AI program uses all three modes in combination, not one or another. Human-in-command establishes the policy envelope. Within that envelope, high-risk decisions route through HITL, high-volume routine decisions route through HOTL, and each mode has its own documentation trail. Confusion arises when sponsors describe their entire AI governance as “human in the loop” when what they actually mean is a mix of program-level authorization (human-in-command) and asynchronous monitoring (HOTL) with only the highest-risk decisions receiving synchronous HITL review. The vocabulary matters because inspectors are increasingly precise about it, and because internal audit teams need to be able to test the specific oversight design that a given use case is claiming.

The distinction matters because regulators, when they use “human in the loop,” typically mean the HITL sense. When they say “human oversight,” they often mean the broader combination of HOTL and human-in-command. Sponsors that read every regulatory phrase as strict HITL end up with unworkable synchronous review of every AI output; sponsors that treat every regulatory phrase as generic “human oversight” end up with an inspection finding.

The FDA/EMA Joint AI Principles (January 2026)

On 14 January 2026, the FDA and the European Medicines Agency jointly published ten guiding principles of good AI practice in drug development.⁷ The joint publication is a genuine regulatory milestone. It is the first time the two agencies have aligned on a foundational AI framework covering the full medicines lifecycle, and it lays the groundwork for future binding guidance in both jurisdictions.⁸

The ten principles cover: human-centric design, a risk-based approach to development and use, adherence to relevant standards, clear context of use, multidisciplinary expertise, robust data governance and documentation, model design and development practices, risk-based performance assessment, life-cycle management, and clear communication of essential information about the AI.⁹

Human oversight appears explicitly in Principle 1 (human-centric design) and implicitly runs through several others, particularly the multidisciplinary expertise principle and the life-cycle management principle. Read together, the ten principles are the joint answer to the question of what human oversight is for: it is the mechanism by which the AI’s outputs remain traceable to human decision-makers who accept accountability for them.

What the Joint Principles Actually Require

The joint principles are not binding law, and neither agency has committed to specific implementation timelines. They function as a shared reference framework. Sponsors preparing for future binding guidance should read them as an early signal of what enforceable requirements will look like.

The human-centric design principle in particular resolves an ambiguity that had been present in earlier documents. It clarifies that human oversight is not a bolt-on control layer added at deployment; it is a design property of the AI system that must be established at the specification stage and validated throughout the lifecycle.¹⁰ If a sponsor cannot show at the concept stage how oversight will operate, later attempts to add human review to a black-box output are not going to satisfy a well-prepared inspector.

The multidisciplinary expertise principle answers a related question about who does the oversight. The principles explicitly call for the integration of data scientists with clinical, regulatory, and quality leads throughout the AI lifecycle. This is not a staffing recommendation; it is a signal that inspectors will expect to see evidence that the humans overseeing the AI understand both the domain and the algorithm’s limits.

The life-cycle management principle answers the timing question. Human oversight cannot be limited to a pre-deployment validation exercise; it must persist across the full operating life of the AI system, including post-deployment monitoring, periodic re-validation, and controlled retirement. This aligns with the EMA reflection paper’s insistence that human oversight is a continuous property of AI use, not a one-time gate. Sponsors that treat validation as the endpoint of oversight will find themselves misaligned with the joint principles’ explicit lifecycle framing.

The clear communication of essential information principle answers a transparency question that had previously been treated inconsistently. It requires that sponsors communicate the AI’s context of use, its known limitations, and the residual uncertainty in its outputs in ways that downstream users can act on. The principle does not mandate full explainability of every model output, which for many modern architectures is not achievable, but it does require that the users of AI outputs have enough information to exercise oversight competently. The line between “explainability” and “interpretability” that the EMA drew in 2024 is preserved and reinforced.

EU AI Act Article 14 and Annex 22: The Binding Requirements

Where the FDA/EMA joint principles are guidance, the EU AI Act is enforceable law. Article 14 of the Act is the binding human-oversight requirement for high-risk AI systems, and high-risk includes essentially all AI used for pharmaceutical decision-making that touches patient safety or product quality.³

Article 14 in Practical Terms

Article 14 requires that providers of high-risk AI systems design and develop those systems in a way that enables effective human oversight during use. The specific capabilities that the AI system must enable include:¹¹

• Understanding capacities and limitations. The overseer must be able to understand the relevant capacities and limitations of the AI system and duly monitor its operation, including in view of detecting and addressing anomalies, dysfunctions, and unexpected performance.
• Awareness of automation bias. The overseer must be aware of the possible tendency to automatically rely on the output produced by the AI system, particularly for systems used to provide information or recommendations to natural persons.
• Correctly interpreting outputs. The overseer must be able to correctly interpret the AI system’s output, taking into account the characteristics of the system and the interpretation tools and methods available.
• Deciding not to use the output. The overseer must be able to decide, in any particular situation, not to use the AI system or to otherwise disregard, override, or reverse the output.
• Intervening or interrupting. The overseer must be able to intervene in the operation of the AI system or interrupt it through a stop button or similar procedure that allows the system to come to a halt in a safe state.

Read literally, these are five distinct capabilities that the AI system, its interface, and the training program for overseers must be designed to enable. A sponsor that deploys a high-risk AI in the EU without evidence that all five are supported is out of compliance, regardless of what its internal policies say about human oversight.¹²

How Annex 22 Extends the Requirements for GxP AI

The EU AI Act sets the horizontal baseline. For pharmaceutical manufacturing specifically, EU GMP Annex 22, published for stakeholder consultation on 7 July 2025, adds sector-specific expectations for AI systems in GMP environments.¹³ Annex 22 operates in tandem with Annex 11: Annex 11 governs the computerized system that hosts the AI, and Annex 22 governs the AI model and its lifecycle within that system.¹⁴

Annex 22 focuses on AI and machine-learning models that support or automate GxP-relevant decisions and activities and that are integrated into the computerized-system landscape governed by Annex 11. For human oversight specifically, Annex 22 reinforces that the person accepting an AI-informed decision must have the competence to evaluate the decision, that the process for that acceptance must be documented, and that periodic review of AI-model performance is a mandatory lifecycle activity.¹⁵

Human Oversight Across the AI Lifecycle

Regulators consistently frame their expectations lifecycle-wise, not system-wise. The oversight required at model development is not the same as the oversight required at validation, deployment, or ongoing monitoring. The joint FDA/EMA life-cycle management principle and the EMA reflection paper’s lifecycle framing both make this explicit.²

A Decision Framework: Synchronous, Asynchronous, or Sampled Review

The practical question a sponsor faces for any AI use case is not “should there be human oversight?” but “what form of human oversight is proportionate to the risk?” The answer determines whether the workflow is feasible at operational scale.

Three review modes are typically available, and matching them to use cases is where governance work actually happens:

Two rules apply across all three modes. First, the selected mode must be justified in the AI risk assessment against the specific consequences of an undetected AI error. Second, the mode must be operationally feasible; sponsors that specify synchronous review of tens of thousands of events per week are creating oversight theater that will fail under load and will be recognized as such at inspection.¹⁶

Applying the Framework to Pharmacovigilance

Pharmacovigilance is the domain where the framework gets its clearest test, because case volumes make purely synchronous review infeasible for most sponsors. The FDA, in its work on AI for individual case safety reports submitted to FAERS, has been explicit that current AI performance requires a human-in-the-loop to ensure quality but has not required that every case receive synchronous review.¹⁷

A defensible tiered approach: serious and unlisted events receive synchronous HITL review; non-serious listed events with high AI-classification confidence route through HOTL with alarm thresholds for classification uncertainty or unexpected patterns; low-risk routine coding tasks are subject to sampled audit against statistically justified sample sizes. Each tier has documented acceptance criteria, and the tier boundaries themselves are subject to human-in-command decision at the program level.¹⁸

The same tiered logic extends to manufacturing. Real-time process control adjustments within a validated design space can operate under HOTL provided the alarm thresholds are properly established and the intervention protocol is documented. Batch release recommendations that translate into regulatory-record decisions require synchronous HITL: a qualified person accepts or rejects, and the acceptance decision itself is the record. Retrospective trend analysis on non-critical parameters may be safely handled through sampled audit. The unifying principle is that oversight intensity scales with the reversibility and consequence of the action the AI is influencing, not with the technical sophistication of the AI itself.

Clinical operations offers a third useful test. AI-driven patient-eligibility screening for clinical trials can operate through HOTL with sampled physician audit of rejections, provided the model’s negative-predictive value has been established on a representative validation cohort and the sampling protocol is statistically justified. AI-generated protocol content, by contrast, requires synchronous HITL by a qualified medical or biostatistical reviewer before it becomes part of a regulatory submission. The distinction is not about which task is harder; it is about which task produces a record that binds the sponsor.

Reviewer Competencies: What “Qualified” Actually Means

Article 14 requires that the humans doing oversight be able to understand the AI’s capabilities, its limitations, and its outputs. This is where a lot of pharma AI governance is thinnest. It is comparatively easy to write a policy that says “qualified reviewer” and comparatively hard to specify what that qualification actually consists of.

Reviewer competency for a high-risk AI system spans three domains, and evidence that a reviewer meets requirements in only one or two of the three is not sufficient:

Domain Expertise

The reviewer must have the substantive expertise to evaluate the AI’s output on its merits. A pharmacovigilance case-assessment AI is overseen by a qualified safety professional. A clinical-endpoint derivation AI is overseen by a qualified medical or biostatistical reviewer. Domain expertise is the traditional pharma competency and is generally the easiest to document.

AI Literacy

The reviewer must have sufficient understanding of how the specific AI system works to interpret its outputs correctly and to recognize failure modes. This includes an understanding of the training data’s provenance and limits, the model’s known error patterns, and the metrics that indicate model uncertainty. AI literacy is not the same as technical fluency in machine learning; it is applied literacy for a specific model.¹⁹

Awareness of Automation Bias

Article 14 explicitly requires that overseers be aware of their own tendency to defer to AI outputs. This is a training requirement, not a system-design requirement. Reviewer training programs must include content on cognitive biases in human-AI interaction, and organizations should evaluate reviewer performance in ways that surface whether reviewers are actually engaging with outputs rather than approving by default.²⁰

The Automation Bias Problem Nobody Wants to Talk About

Automation bias is the well-documented human tendency to over-rely on automated outputs and to under-detect errors that the automated system makes. The evidence base on it in healthcare AI is substantial and consistent: reviewers frequently miss AI errors, particularly in high-throughput or time-pressured settings, and confidence in AI outputs increases with exposure regardless of whether performance justifies it.²¹

The reason regulators wrote automation bias awareness into Article 14 is that they have read the same literature. Human-in-the-loop review can, if badly designed, actively reduce the quality of decisions relative to human-only review, because the AI’s presence induces a deference that undermines the reviewer’s critical evaluation.²²

The design implications are concrete:

• Progressive disclosure. Interface design should require reviewers to form an independent judgment on the underlying data before the AI’s recommendation is displayed, at least on a defined sampled basis. This is one of the more effective countermeasures in the empirical literature.
• Forced deliberation. For consequential decisions, the interface should require the reviewer to actively record the reasoning for accepting or rejecting the AI output, not simply click an approve button. The record itself functions as evidence of engagement and as a source for retrospective quality review.
• Controlled disagreement in training. Reviewer training should include cases where the AI is deliberately wrong, with feedback on the reviewer’s detection. Ongoing quality programs can include unannounced injected cases to monitor whether reviewers are engaging.
• Confidence signaling. AI outputs should communicate the model’s uncertainty in ways that reviewers can act on. A recommendation with 60% model confidence should look different from one at 99% confidence, and the interface should not display them identically.

Implementation: Making Human Oversight Actually Work

Compiling the regulatory expectations into an implementation approach requires decisions in six areas. Sponsors that have made those decisions explicitly, and can show inspectors the evidence, are in a substantially better position than sponsors that have written policies but not translated them into workflow.

1. Risk Classification of Each AI Use Case

Every AI use case in a GxP environment should be classified against the consequences of an undetected error. The risk classification determines the oversight mode (synchronous, asynchronous, sampled), the reviewer competency requirements, and the validation rigor. GAMP 5 Second Edition and the joint FDA/EMA principles both anchor their frameworks in risk-based classification, and inspectors will expect to see the classification methodology.²³

2. Written Oversight Specification per AI System

Each production AI should have a documented oversight specification: what output is being overseen, who oversees it, at what point in the workflow, against what criteria, with what documented evidence. This is the artifact that makes Article 14 auditable. Its absence is what turns a plausible policy into an inspection finding.

3. Reviewer Competency Matrix

For each AI system, define the domain expertise, AI literacy, and automation-bias-awareness requirements the reviewer must meet, and maintain training records to demonstrate that assigned reviewers actually meet them. The matrix should be updated when the AI system is modified, because AI-literacy requirements can change materially with model updates.

4. Interface Design Aligned to Oversight Objectives

The AI’s user interface is a component of the oversight system. Interfaces that make it easy to approve without engaging are creating audit exposure. Design reviews should include human-factors expertise, particularly for high-risk applications, and interface changes should be considered validation-relevant.

5. Ongoing Monitoring with Human Review Gates

Production monitoring must include the human review activities themselves, not just AI-performance metrics. Track reviewer agreement rates with AI outputs, time-to-decision distributions, and outcomes on injected quality-assurance cases. These metrics detect automation-bias drift before it produces an inspection finding.

6. Governance Committee with Real Authority

Human-in-command oversight requires a governance body that can make binding decisions about AI deployment, retirement, and modification. The body should include representation from clinical or scientific, quality, regulatory, data science, and IT. Its charter, decision authorities, and meeting records are inspection-relevant documentation.

The specific test of whether a governance body is real, rather than performative, is whether it has ever refused an AI deployment request or forced retirement of a deployed system. Bodies that only approve, and only ratify decisions already made elsewhere, are governance in name only and will be recognized as such. The health of an AI governance program is best measured not by how many systems are deployed but by how many were rejected, delayed, or scoped down as a result of the governance review, and by how quickly identified performance issues in deployed systems triggered documented action.

What Inspection Readiness Actually Looks Like

An inspection-ready AI oversight program should be able to produce, on demand, the following artifacts for any deployed AI system: the risk classification and its rationale; the oversight specification identifying mode, reviewer, review point, and acceptance criteria; the reviewer competency records for the individuals actually performing oversight; validation evidence including performance on representative subgroups and bias analysis; monitoring evidence showing that both AI performance and reviewer behavior are being tracked; the change-control record for any modifications since deployment; and the governance-body approval and periodic-review records.

None of these artifacts are individually novel. Pharma quality systems have been producing analogous artifacts for computerized systems for decades. What is new is the integration of AI-specific evidence into those artifacts and the explicit framing of oversight as an ongoing lifecycle activity subject to periodic effectiveness review. Sponsors that already have strong CSV programs will find that most of the infrastructure they need already exists; the work is in extending it to cover the AI-specific dimensions, not in building parallel systems.

Conclusion

The regulatory direction on human oversight of pharmaceutical AI is now clear enough to plan against, even where the specific rules are still consolidating. The FDA and EMA have aligned on ten joint principles that anchor the field in human-centric, risk-based design. The EU AI Act’s Article 14 makes five specific oversight capabilities binding for high-risk systems. Annex 22 extends those expectations into GMP-regulated AI. GAMP 5 Second Edition and the standalone ISPE GAMP AI Guide translate the principles into validation practice. The consistent message across all of these is that “human in the loop” is not a slogan; it is a set of design, training, and documentation decisions that must be made at the system level, justified by risk, and evidenced through the lifecycle.

The practical work sponsors face is neither philosophical nor optional. Choose the review mode (synchronous, asynchronous, or sampled) that is proportionate to the risk of each use case. Document the reviewer competencies at the intersection of domain, AI literacy, and automation-bias awareness. Design interfaces that force engagement rather than reward deference. Monitor the human review activity itself, not just the AI. Convene governance with real authority to authorize, modify, and retire AI systems as their performance evolves.

Sakara Digital works with pharma and biotech organizations building the oversight architectures that make AI deployments defensible under FDA, EMA, and EU AI Act scrutiny. If you are translating the joint principles into concrete workflows, structuring reviewer competency programs, or preparing an AI-enabled system for inspection, and want an independent perspective on where to start, we are happy to have that conversation.