Red Teaming Pharma AI Models: A Structured Test Plan

Validation and Red Teaming Answer Different Questions

A validation package is an argument. It says: here is what we intended the system to do, here is the evidence that it does that, here is the evidence that the people running it are trained and the records it produces are attributable, legible, contemporaneous, original, and accurate. Every element is built around intended use. That is not a weakness. Intended use is what makes computerized system validation tractable in the first place, and it is why the discipline has held up across four decades of technology change.

Adversarial testing starts from the opposite end. It does not ask what the system is for. It asks what the system can be made to do. The tester is not verifying a requirement; the tester is looking for behavior nobody wrote a requirement about, because nobody imagined it. A pass on a validation script means the system met a stated expectation once, under controlled conditions, with data the test author chose. A red team finding means the system did something no one expected, under conditions someone deliberately constructed.

The distinction matters more for AI systems than for the deterministic software that GxP validation grew up around. A rules engine that passes its test cases will behave the same way on the same input forever. A language model will not necessarily do that, and even a static model whose weights are frozen will produce very different behavior across inputs that look superficially similar. The input space is effectively unbounded, so the coverage argument that underpins traditional operational qualification does not carry over. You cannot test the whole space. You can only test the parts of it you deliberately choose, which makes the choice of what to test the entire game.

The three claims validation cannot make about an AI system

It is worth being precise about what a passing validation package does not establish, because this is where most internal debates get stuck.

It does not establish behavior on inputs unlike the qualification data. If the performance qualification used 200 representative documents, the evidence covers those 200 documents and, by argument, the population they were drawn from. It says nothing about a document from a new supplier with a different template, a scanned page with poor optical character recognition output, or a submission in a language the model saw rarely during training.

It does not establish behavior under instruction. Traditional software does not take instructions from its data. A language model does, and that is not a bug in a particular product. It follows from the architecture: instructions and content arrive through the same channel, and the model has no reliable way to tell one from the other. A validation script that feeds the model a clean document proves nothing about a document with a paragraph of text aimed at the model rather than the reader.

It does not establish that the system knows when it is wrong. Validation measures accuracy against a reference. It rarely measures calibration, which is whether the confidence the system expresses matches how often it is actually right. A model that is 85% accurate and expresses uniform high confidence is far more dangerous in a regulated workflow than a model that is 75% accurate and reliably flags the cases it is unsure about, because the second one lets a human reviewer allocate attention.

What the Frameworks Actually Ask For

Before writing a plan it helps to know what the published frameworks require, because the answer is narrower than vendors suggest and broader than most quality organizations assume.

The NIST AI Risk Management Framework organizes activity into four functions: Govern, Map, Measure, and Manage.¹ Adversarial testing lives mainly in Measure, alongside the more familiar performance metrics. The Generative AI Profile, published as NIST AI 600-1 in July 2024, goes further and names AI red teaming explicitly as a suggested action across twelve named risk categories, including confabulation, information integrity, information security, data privacy, and human-AI configuration.² The Profile is a voluntary companion document, not a regulation, but it is the most detailed public statement of what a structured adversarial program is supposed to cover, and it is the vocabulary regulators are converging on.

NIST also publishes the taxonomy that gives the attack classes their names. NIST AI 100-2 E2025, released in March 2025, is a taxonomy and terminology of adversarial machine learning attacks and mitigations.³ It splits systems into predictive AI and generative AI and enumerates evasion, poisoning, and privacy attacks for both, with misuse attacks added for generative systems.⁴ The 2025 edition expanded the earlier version to add clean-label poisoning and indirect prompt injection, which tells you something about how quickly the threat picture has moved. If you want one document to anchor the naming convention in your test plan so that findings can be compared across years, this is it.

On the regulatory side, Article 15 of the EU AI Act requires high-risk systems to achieve an appropriate level of accuracy, robustness, and cybersecurity and to perform consistently in those respects across the lifecycle. It specifically calls for technical measures addressing data poisoning, model poisoning, adversarial examples designed to make the model make a mistake, confidentiality attacks, and model flaws.⁵ Article 72 then makes this a continuing obligation by requiring post-market monitoring that verifies performance holds after deployment.⁶ For providers of general-purpose models with systemic risk, Article 55 is more direct still: model evaluation using standardized protocols, including conducting and documenting adversarial testing to identify and mitigate systemic risk.⁷

Most pharmaceutical organizations are deployers rather than providers of general-purpose models, so Article 55 will not bind them directly. It matters anyway, for two reasons. It sets the reference standard a supplier will be measured against, which makes it a reasonable basis for supplier questions. And it establishes adversarial testing as a documented, reportable activity rather than an informal one, which is the direction sectoral guidance tends to follow.

Two more documents belong in the frame. The FDA draft guidance on the use of artificial intelligence to support regulatory decision-making for drug and biological products, issued in January 2025, proposes a risk-based credibility assessment framework built around a defined context of use, with model risk derived from model influence and decision consequence.⁸ That framing is directly usable for red team scoping: the systems where an adversarial finding matters most are the ones where the model has high influence over a decision with serious consequences. And the draft EU GMP Annex 22, published for consultation in July 2025, brings the same logic into manufacturing, with restrictions on which model types may be used in applications that touch product quality or patient safety.

What none of these documents provides is a test plan. They tell you adversarial testing is expected. They do not tell you what a test case looks like, who writes it, how often you run it, or what you do with a finding. That is the work.

The Attack Classes, With Pharma Test Cases

A test plan built around vague categories like “security” or “misuse” produces vague findings. Build it around attack classes instead, with concrete test cases written against your actual system. Four classes cover most of what matters for the systems pharmaceutical and biotech organizations are actually deploying.

1. Direct prompt injection and instruction override

The simplest class. A user types something intended to override the instructions the system was given. Prompt injection has held the top position in the OWASP Top 10 for large language model applications across two editions, and the reason is structural: the model processes instructions and data through the same channel with no reliable separation, so input crafted to look like an instruction is frequently treated as one.⁹

Test cases for this class are cheap to write and should be the first thing any team runs, because they establish a baseline. Ask the system to reveal its system prompt. Tell it the previous instructions were a test and the real task is different. Tell it you are the system administrator and the safety rules have been suspended for maintenance. Ask it to summarize its own configuration. Wrap the override in a format the model treats as authoritative, such as a fake configuration block or a message that looks like it came from the platform rather than the user.

What you are looking for is not just whether the override works. It is what the system does when it works. A chatbot that reveals its system prompt is embarrassing. A system that reveals its system prompt and then also exposes the connection string to the document store behind it is a different severity entirely.

2. Indirect prompt injection through ingested content

This is the class that matters most for pharmaceutical and biotech systems, and it is the one most often missing from a first-draft test plan. A model that reads documents is attackable by whoever writes the documents.

The foundational research on this is by Greshake and colleagues, presented at AISec in 2023, which showed that applications built on top of language models blur the line between data and instructions in a way that lets an attacker plant instructions in content the system will later retrieve.¹⁰ The attacker never touches the application. They write a document, and the document does the work when the system ingests it. The paper demonstrated the technique against production systems, not just laboratory prototypes.

Now map that onto what pharmaceutical organizations are actually building. Systems that read supplier qualification packets. Systems that summarize literature for signal detection. Systems that triage adverse event narratives arriving from call centers and patient support programs. Systems that ingest deviation reports and propose classifications. Systems that read contract manufacturing batch documentation. Every one of these takes content from outside the organization and passes it to a model with instructions attached.

Concrete test cases for this class, written against your ingestion pipeline rather than the model in isolation:

• Hidden text in a supplier document. Place instructions in white-on-white text, in a zero-height table row, and in PDF metadata fields. Confirm what your extraction step actually passes through. Most teams are surprised.
• Instructions inside a literature abstract. If a signal detection or literature surveillance system ingests abstracts from an external source, test whether text inside an abstract can change how the system treats the record, suppress it from a summary, or alter a classification.
• Instructions in a free-text patient narrative. Adverse event intake takes unstructured text from people outside your organization. Test whether narrative content can influence seriousness assessment or expectedness determination.
• Instructions in a filename or folder path. Frequently overlooked, and frequently included in the context the model receives.
• Chained injection. Content that instructs the model to write something into a downstream record, which is then read by a second system. This is where an agentic architecture turns a single bad document into a propagating problem.

The mitigation question follows directly from the test result. If hidden text survives extraction, the fix is often in the pipeline rather than the model: strip formatting, normalize to plain text, flag documents whose extracted text differs materially from their rendered text. That is a satisfying finding, because it produces an engineering change with a clear verification path. The joint international guidelines for secure AI system development make the same point at the design stage, treating input handling and supply chain content as security-relevant surfaces rather than plumbing.²⁰

3. Jailbreaks that bypass a safety instruction

A jailbreak is an input that gets the model to do something its instructions prohibit. In consumer contexts the discussion is about harmful content. In a pharmaceutical context the prohibitions that matter are usually different and more specific: do not provide dosing advice, do not speculate about causality in an individual case, do not generate content that would constitute promotional claims, do not answer questions about products outside the approved label, do not produce a regulatory conclusion without citing source documents.

These constraints are usually implemented as instructions in a system prompt, which means they are exactly as durable as the model’s willingness to follow instructions in the presence of contrary input. Test that assumption rather than trusting it. Standard techniques include role framing (asking the model to act as a fictional character not bound by the rules), hypothetical framing (asking what it would say if it were allowed), incremental escalation across a multi-turn conversation, translation into another language and back, and encoding the request so the guardrail’s pattern matching does not fire.

There is direct evidence that this class of testing finds real problems in clinical settings. A red teaming study published in npj Digital Medicine convened clinicians, students, and technical staff to stress-test a widely used model against real-world clinical cases, categorizing inappropriate responses along axes of safety, privacy, accuracy, and bias. Roughly one in five responses across the exercise was unsafe or contained bias, and the team found dozens of vulnerabilities in a newer model version that were not present in the older one.¹¹ That last detail is the operationally important one, and we return to it when we get to the cadence question.

4. Data extraction and cross-tenant leakage

This class asks whether the system can be made to reveal information it should not. There are three distinct versions worth separating in a test plan.

Training data extraction. Carlini and colleagues demonstrated in 2021 that language models can be prompted to emit verbatim sequences from their training data, including personally identifiable information, using only query access.¹² For an organization that fine-tuned a model on internal documents, this is not an abstract concern. If safety narratives, unredacted case records, or confidential development data went into a fine-tuning set, the question of whether they can come back out is a real one with a testable answer.

Retrieval scope violation. More common, and easier to find. If a system retrieves from a document store with access controls, test whether a user in one role can construct a query that surfaces content from another role’s scope. Test whether the model summarizes a document the user could not open directly. Test whether citations reveal the existence and titles of documents the user is not cleared to see, which is itself a disclosure.

Cross-tenant leakage. Where a shared platform serves several business units, several study teams, or several partner organizations, test whether content from one boundary appears in another. This is the question worth asking a supplier directly, and it is the one where a specific test result is far more useful than a policy statement in a questionnaire.

Sensitive information disclosure sits at position two in the OWASP list for a reason, and the failure modes named there include personally identifiable information surfacing in interactions and proprietary logic exposed through poorly configured output.¹³

Input manipulation across a decision boundary

The fourth class in the NIST taxonomy applies to the predictive models many organizations have quietly run for years, and the test is different in character. Instead of writing text, you perturb an input until the classifier changes its answer. For a model that classifies deviations by criticality, that means constructing pairs of near-identical descriptions that land on opposite sides of the boundary. For a visual inspection model, it means small changes to lighting, orientation, or image compression. For a model that scores site risk or predicts an audit finding, it means finding the smallest change in an input feature that flips the output.

The finding you want here is not usually “the model can be fooled”. It is a map of where the boundary is unstable, because that tells the business where human review has to be mandatory. A boundary that is stable across a wide margin can carry a lighter review burden. A boundary where a two-word change flips the classification cannot.

The Failures More Likely Than Attacks

Here is where most red team programs in regulated industries go wrong. They are staffed by security people, scoped as a security exercise, and aimed entirely at a malicious attacker. That framing will miss the failures most likely to actually occur, because for the majority of pharmaceutical AI systems there is no attacker. There is a busy person, an unusual document, and a model that answers anyway.

Microsoft’s AI red team, reporting on eighty operations covering a hundred generative AI products, made a related point directly: you do not have to compute gradients to break an AI system, and responsible AI harms are pervasive but hard to measure.¹⁴ The team’s summary of its own lessons is worth reading in full, but the operational takeaway is that simple techniques and non-security failure modes account for a large share of what a red team finds.¹⁵ A program that only looks for attackers is looking in the smaller half of the space.

Out-of-distribution input

The most common real failure is the simplest: the system receives something unlike anything in its training or qualification data, and it responds as confidently as it responds to anything else. A systematic review of dataset shift in machine learning for health predictions found that model-based monitoring and statistical testing were the most frequently used detection strategies, with retraining and feature engineering the predominant corrections.¹⁸ The detection half of that finding is the part that belongs in a red team plan, because if the system cannot detect that it has left familiar territory, no correction strategy will trigger.

The external validation result for the sepsis prediction model is the reference case for this failure mode. A model deployed at scale, presumably performing acceptably on the data it was developed against, achieved an area under the curve of 0.63 in an external cohort of 38,455 hospitalizations, identifying a small fraction of the cases that mattered while generating alerts on nearly one in five admissions.¹⁷ Nothing was attacked. The population was simply different from the one the model was built on, and no test in the deployment process was designed to notice.

Test cases for out-of-distribution behavior are straightforward to construct once the team accepts that they belong in the plan:

• Documents from a template the system has never seen, including a poorly scanned one
• A record from a product, therapeutic area, or region outside the qualified scope
• Input in a language the system was not qualified for, including mixed-language documents
• A record with a critical field missing, empty, or filled with a placeholder such as “TBD”
• Data from a period before a reference terminology version change, and after it
• An input at the extreme of the plausible numeric range, and one just outside it

For each, the question is not only whether the answer is right. It is whether the system indicated any reduced confidence, declined to answer, or routed to a human. A system that fails loudly on unfamiliar input is a system you can operate. A system that fails silently is one you cannot.

Ambiguous and contradictory source documents

Regulated organizations run on documents that disagree with each other. A specification says one thing and a change control record says another. A batch record contradicts a deviation investigation. A protocol amendment supersedes a section that a later document still cites. A supplier’s certificate of analysis and the incoming inspection result do not match. Humans handle this by noticing the conflict, escalating it, and getting a decision. Models frequently handle it by picking one and moving on, without indicating that there was a choice to make.

This is one of the highest-value test areas in the whole plan and it requires no security expertise at all. Construct document pairs that contradict each other on a material point and feed both to the system. Vary which one appears first, since order frequently drives the answer. Vary which one is longer or more detailed. Include a case where one document supersedes the other but the supersession is stated only in a header. Then evaluate on a single criterion: did the system surface the conflict, or did it resolve the conflict silently?

The confidently wrong answer

NIST calls it confabulation. Most people call it hallucination. Whatever the label, the operational problem is that the failure comes packaged in the same fluent, well-structured, plausible prose as a correct answer, which strips the reviewer of the cues they normally use to allocate attention.

Fabricated citations are the cleanest measurable version of this failure, and the numbers are not marginal. A controlled study of citation generation found that roughly one in five citations produced across six simulated literature reviews was entirely fabricated, with topic familiarity and prompt specificity both affecting the rate.¹⁶ For any workflow where a model drafts text that will be reviewed by someone who trusts a citation because it looks like a citation, that is the number to put in front of the process owner.

There is also an adversarial version of this failure that sits at the boundary between the two halves of the plan. A multi-model assurance analysis published in Communications Medicine found that language models used for clinical decision support were highly vulnerable to adversarial hallucination attacks, where fabricated details planted in the input are picked up and elaborated by the model as though they were real.¹⁹ In a pharmacovigilance or medical information context, that is a plausible route from one incorrect input to a chain of confidently stated fiction in a record that goes on to be relied upon.

Test cases for the confidently wrong answer follow a pattern. Ask questions whose correct answer is that the source does not say. Ask about a study, a product, or a guideline that does not exist and see whether the system invents one. Plant a false but plausible detail in the input and see whether it is repeated, elaborated, or challenged. Ask the same question ten times and measure how much the answer varies, since instability across identical inputs is itself a finding worth logging. And in every case, capture the confidence the system expressed, because the gap between confidence and correctness is the number that determines how much human review the workflow needs.

Writing the Test Plan: Structure and Test Case Format

A red team plan that lives as a slide deck will not survive contact with a quality system. Write it as a controlled document with the same sections a validation plan has, because that is the format your organization already knows how to review, approve, and store.

The test case format

Keep it to eight fields. Anything longer will not be filled in consistently, and consistency is the entire value.

Defining expected safe behavior before running the test is the discipline that separates a red team from an exploratory session. It is easy, after seeing a clever output, to talk yourself into accepting it. Writing the acceptance criterion first removes that option.

Severity, Scoring, and What Counts as a Finding

A red team that reports “the model can be jailbroken” has not given the business anything to act on. Severity has to be derived from consequence in the workflow, not from how impressive the technique was.

The most workable approach uses two axes borrowed from quality risk management and the FDA credibility framing. The first is reachability: how plausibly does this input arise in real operation? An attack requiring privileged access to the vector database is less reachable than a hidden paragraph in a supplier PDF, which anyone in the supply chain can produce. The second is consequence: what happens downstream if the system behaves this way and nobody catches it? A wrong summary that a reviewer signs is different from a wrong summary that a reviewer reads alongside the source document.

The Operating Model: Who Runs It and How Often

This is the section that determines whether the program exists in eighteen months.

Independence

The people who built the system cannot be the people who red team it. This is not a comment on anyone’s integrity. It is the same principle that keeps quality assurance separate from manufacturing. Builders know the intended path and their attention naturally follows it. They also have an interest, however unconscious, in the system passing. The EU AI Act’s provisions on adversarial testing for systemic-risk models anticipate the involvement of independent external experts for exactly this reason.⁷

Independence does not require a large dedicated team. In most mid-size pharmaceutical and biotech organizations a workable structure is a standing group of four to six people drawn from quality, information security, the relevant business function, and data science, none of whom work on the system under test, coordinated by a named owner who is accountable for the schedule and the findings register. External specialists are worth bringing in for the deeper technical classes, particularly extraction and cross-tenant testing, where the techniques move quickly.

The mix matters as much as the independence. The NIST Generative AI Profile distinguishes several types of red team, including expert teams and combined human and automated approaches, and the clinical study cited earlier deliberately combined clinicians, students, and technical staff.¹¹ A security specialist will find the injection paths. Only a pharmacovigilance or quality professional will recognize that a particular output would have produced the wrong seriousness assessment.

Cadence and triggers

Annual testing tied to periodic review is the floor for any system in scope. It is not sufficient on its own, because the thing being tested changes faster than the review cycle. Define unscheduled triggers explicitly in the plan:

• Model version change. The single most important trigger. The clinical red teaming study found dozens of vulnerabilities in a newer model version that did not exist in the older one.¹¹ An upgrade is not automatically an improvement in the dimensions you care about, and a supplier’s release notes will not tell you which of your test cases now fail. Rerun the regression set.
• System prompt or instruction change. If the guardrails are instructions, editing them is editing a control.
• New data source or new document type ingested. Every new input channel is a new party who can write into your context window.
• Scope expansion. A new user population, a new therapeutic area, a new region, or a new decision the output feeds.
• Retrieval configuration change. Chunking, embedding model, index permissions, or ranking changes all alter what reaches the model.
• A relevant external finding. A published technique or an industry incident affecting the same architecture should trigger a targeted round, not a note in a risk register.

Where findings go

A finding that lands in a shared spreadsheet will be reviewed once. Route findings into the quality system, using the routes that already exist. A critical finding on a system in production is a deviation. A finding that indicates a systemic weakness gets a CAPA. A finding that requires a change to the model, the prompt, the pipeline, or the procedure goes through change control. A finding accepted without remediation gets a documented risk acceptance signed by someone with the authority to accept it.

This is not bureaucratic overhead. It is the only route by which a finding reliably gets an owner, a due date, and evidence of closure, and it is the only route that produces a record an inspector will recognize. Organizations that keep red team findings outside the quality system usually do so because they are worried about what happens when an inspector sees them, which brings us to the last question.

Documentation and the Inspection Question

Every organization that starts this work eventually asks the same question, usually in private: if we write these findings down, are we creating evidence of a known weakness that will be used against us?

The honest answer is yes, you are creating a record of a known weakness. That is what a finding is. The answer to the concern behind the question is that the alternative is worse in every direction. Not testing does not mean the weakness is absent. It means it is undocumented, unassessed, and uncontrolled, and that the first person to characterize it will be someone outside your organization. In a system built on risk-based quality management, a documented weakness with an assessment and a control is an ordinary artifact. An undocumented one is a gap in the quality system.

The way to handle this is to decide the documentation rules before the first round runs, not after the first uncomfortable finding.

Decide these five things in advance

• What is recorded. Findings, severity, rationale, and closure evidence. Raw exploratory transcripts do not all need to be controlled records, but the plan must state which artifacts are controlled and which are working papers, and it must be consistent. Deciding case by case after the fact is what creates a problem.
• Where it is recorded. The findings register and its relationship to the quality system. One register per system, retained for the life of the system plus the applicable retention period.
• Who risk-assesses. A named role, not the tester. Use the risk management process the organization already has rather than inventing a parallel one.
• What closure requires. Verification that the fix works, which means rerunning the test case, not a statement that a change was made. Closure without a rerun is not closure.
• What is produced on request. Agree in advance with quality and legal what an inspector or auditor is shown. In most cases the honest answer is the plan, the register, the risk assessments, and the closure evidence, presented as part of the system’s lifecycle documentation.

Presented that way, a red team file is a strength. It shows a documented method, independent execution, risk-based prioritization, and closed actions. That is the same story a good deviation trend or a well-run internal audit program tells. The organization that has none of it, and is asked how it knows its language model cannot be instructed by a supplier document, has a harder conversation.

What good looks like after a year

An organization a year into a working program has a small number of specific things. A controlled red team plan per AI system in scope, sized to the system’s risk. A regression suite that grows with every round. Two or three completed rounds on the highest-risk systems, with findings that went into the quality system and came back out closed. A supplier questionnaire that asks about adversarial testing results rather than adversarial testing policies. And a defined trigger list that has actually fired at least once, which is usually the moment the program proves it is real.

What that organization does not have is a guarantee. Adversarial testing does not prove absence of weakness, and a plan claiming otherwise is worse than no plan. What it produces is a bounded, evidenced statement about where the system was probed, what was found, and what was done. In a regulated environment, that is what a control looks like.

Conclusion

The reason red teaming is hard for pharmaceutical and biotech organizations is not technical. It is that the discipline runs against the grain of how regulated quality thinking works. Validation is built on stating an intended use and proving conformance to it. Adversarial testing is built on assuming the statement of intended use is incomplete and going looking for what it left out. Those are genuinely different habits of mind, and asking the same team to hold both is asking a lot. The organizations that make progress are the ones that stop treating red teaming as an extension of validation and give it its own plan, its own independent people, its own cadence, and its own route into the quality system.

The second thing that separates programs that work from programs that stall is where they aim. Attack classes are the part that gets attention, and indirect prompt injection through ingested content deserves every bit of the attention it gets, because a model that reads documents is attackable by whoever writes the documents and pharmaceutical organizations read documents written by an enormous number of external parties. But the failures most likely to occur next quarter are the quiet ones: a document unlike anything in the qualification set, two source records that disagree, and an answer that is wrong and sounds exactly like an answer that is right. A red team that only hunts attackers will file a clean report and miss all three.

Sakara Digital works with pharma and biotech organizations building the governance and testing practices behind production AI systems, including red team plans that quality organizations can actually run and defend. If you are standing up adversarial testing for a system already in use, or trying to work out which of your AI systems warrant it first, we are happy to have that conversation.

For Further Reading