Validating an AI Agent That Calls Other Systems: Scope, Boundaries, and Logs

Why an Agent That Acts Is a Different Validation Problem

An AI agent, in the sense used here, is a language model given instructions and a set of tools, running in a loop. It receives a request, decides which tool to call, reads what comes back, and decides what to do next until it considers the task done. The tools are the part that matters for validation. A tool might query the deviation module of an eQMS, pull out-of-specification history from a LIMS, create a draft change request, assign a training item, or file a document into a controlled folder.

Compare that with the generative AI tools most quality teams have looked at so far. A drafting assistant writes text, and a person reads it, edits it, and decides what to do with it. The person is the one who acts. With an agent, the output is the action. The model’s choice becomes a record in a system that an inspector may later pull.

Three Things Change When the Output Is an Action

The object of validation gets bigger. You are no longer validating a model and its prompt. You are validating a chain: the model, its instructions, the tool code, the identities and permissions the tools run under, the target systems the tools touch, and the orchestration code that ties it together. A defect in any link produces a bad record.

Behavior depends on the path. The same request can lead to a different sequence of tool calls on a different run. A traditional interface between a LIMS and an ERP does the same thing every time it receives the same message. An agent may query two systems in one order today and in a different order tomorrow, and may decide it needs a third lookup that it skipped yesterday. Your test evidence has to account for that variation.

Data becomes instructions. Whatever a tool returns goes back into the model as input. If a deviation description, a supplier email, or a document field contains text written to steer the agent, the agent may follow it. NIST’s Center for AI Standards and Innovation (CAISI) calls this agent hijacking and defines it as a type of indirect prompt injection in which an attacker plants instructions in data the agent will read, causing it to take unintended actions.⁸ A validated interface that moves data from one field to another has no equivalent weakness.

What the Evidence Says About Agent Reliability

Published research on agents that use tools gives a sober picture. None of it comes from GxP settings, but the patterns are general enough to shape how you design tests.

The τ-bench study tested agents on tasks that required following domain rules while talking to a simulated user and calling tools. The authors reported that even state-of-the-art function-calling agents such as gpt-4o succeeded on fewer than half of the tasks, and that consistency across repeated attempts was poor.⁶ The AgentDojo benchmark, which includes 97 realistic tasks and 629 security test cases, found that current models fail at many tasks even when no one is attacking them, and that existing prompt injection attacks break some security properties but not all.⁷

Two lessons follow for validation. First, a single successful test run proves little about an agent. Second, the agent’s own judgment cannot be the only thing standing between a bad instruction and a GxP record. Both points come back later in this article.

Where Agents Fit Under Current and Draft Rules

The draft EU GMP Annex 22 on artificial intelligence, published for consultation in July 2025, is direct on this point. It applies to static, deterministic models and states that it does not apply to generative AI and large language models, and that “such models should not be used in critical GMP applications.”¹ For non-critical GMP applications, the draft says qualified personnel should be responsible for making sure outputs are suitable for the intended use (a human-in-the-loop), and that the principles in the annex may be considered where they apply.¹

Annex 22 is still a draft. EMA held a two-day expert workshop on 30 June and 1 July 2026 to gather evidence on how to treat generative AI and large language models, after the consultation suggested support for potentially enabling their use. The event page gives no date for a final text.³ Until it does, the sensible reading for an LLM-based agent is plain: keep it out of critical GMP decisions, give it supporting tasks, and keep a qualified person responsible for what it produces.

That does not mean a supporting agent escapes validation. The systems it writes into are validated computerized systems, and the draft revision of Annex 11 applies to all computerized systems used in manufacturing of medicinal products and active substances.² An agent that creates or changes records in those systems is part of how they are used. FDA’s final Computer Software Assurance (CSA) guidance, reissued on 3 February 2026 for device manufacturers under the Quality Management System Regulation, is not written for drug manufacturers, but its risk-based thinking is a useful reference for pharma teams. Its examples of software functions that are generally high process risk include those that: “Measure, inspect, analyze and/or determine acceptability of product or process with limited or no additional human awareness or review.”⁴ An agent designed to act without a person looking is, by that logic, the high-risk case.

Scope: What the Agent Is For and What It May Not Touch

Validation starts with requirements, and the draft Annex 11 is explicit that this holds no matter how a system is built. It asks regulated users to establish and approve system requirements that describe the functionality they have automated and rely on, whether the system is developed in-house, bought off the shelf, or provided as a service.² It also expects the requirements to cover interfaces and, where relevant, to include process maps and data flow diagrams.²

For an agent, scope has more parts than a typical user requirements specification. Five of them matter most.

1. The Task Boundary

Write down which requests the agent is meant to handle and which it should refuse. “Summarize open deviations for a product and draft a trend note for the site quality lead” is a task boundary. “Help with quality work” is not. The draft Annex 22 asks that the intended use of a model be described in detail, based on in-depth knowledge of the process, including the full range of inputs it will see and any limitations.¹ Even though the annex excludes language models from its formal scope, that is the right standard to borrow. A process subject matter expert, not the development team, should own this description.

2. The Tool Inventory

List every tool the agent can call. For each one, record the target system, the operation (read or write), the data objects it touches, and whether it can create, change, or delete a GxP record. This inventory becomes the backbone of the risk assessment and the test plan.

Tool design has a direct effect on how hard this is. Anthropic’s engineering guidance on writing tools for agents warns that “More tools don’t always lead to better outcomes.” It names a common mistake: tools that simply wrap existing software functions or API endpoints.¹⁵ In GxP terms, a general tool such as call_qms_api(endpoint, body) makes scope impossible to state, because the agent could reach any endpoint the credentials allow. A narrow tool such as create_draft_capa(deviation_id, summary) tells you exactly what can happen. The same guidance recommends grouping related tools under common prefixes (namespacing), which also makes the inventory easier to read, since every tool name shows which system it touches.¹⁵

3. Negative Scope

List what the agent must never do, in plain terms: approve, sign, release, close, delete, change master data, change access rights, or act in a system not on the inventory. Negative scope sounds redundant if the tool inventory is complete, but it earns its place in two ways. It gives testers a list of prohibited outcomes to try to provoke, and it gives reviewers a short statement to check the permission design against.

4. The Configuration

An agent’s behavior is set by its configuration as much as by its code: the model and model version, the system instructions, the tool descriptions the model reads to decide which tool to use, the tool code, and any limits on retries or run length. The draft Annex 11 asks that configurable functionality be described in the requirements and that the chosen configuration be documented in a controlled configuration specification.² For an agent, the tool descriptions belong in that specification. They are text the model follows, and a changed description can change which tool the agent picks.

5. The System Boundary Diagram

Draw it. Show the agent, the orchestration layer, each tool, each target system, each identity, where logs are written, and where human approvals happen. The diagram is the single most useful document for an auditor trying to understand the agent, and drawing it tends to expose connections nobody listed.

Tool Permissions: Least Privilege for a User That Is Not a Person

Once scope is written, permissions put it into force. This is where most of the real control comes from, because a permission enforced by the target system holds no matter what the model decides.

What the Rules Already Say About Accounts and Access

The draft Annex 11 is strict about accounts. It states: “All users should have unique and personal accounts. The use of shared accounts except for those limited to read-only access (no data or settings can be changed), constitute a violation of data integrity.”² It sets two guiding principles for access: segregation of duties, so that users involved in GMP activities do not hold administrative privileges, and least privilege, so that users do not hold more access than their job requires.² In the United States, 21 CFR 11.10 requires controls that include limiting system access to authorized individuals and authority checks to ensure only authorized individuals can alter a record or perform the operation at hand.⁵

None of these texts were written with agents in mind, but they apply cleanly once you ask the obvious question: whose account is the agent using?

Three Identity Patterns, One of Them Unacceptable

The agent borrows a person’s credentials. This is the pattern to rule out. Every record the agent writes appears to come from that person, the audit trail no longer tells the truth about who acted, and the person holds far more access than any single agent task needs. It fails both the unique account principle and basic attribution.

The agent runs under a shared service account. Better, but still weak. If several agents, or an agent and an integration, use the same account with write access, the draft Annex 11 language on shared accounts applies directly, and you lose the ability to say which agent did what.

The agent has its own identity, and the requesting person is recorded alongside. This is the pattern to aim for. Each agent deployment gets a unique account whose rights match its tool inventory and nothing more. The person who asked for the work is captured in the agent’s action log and, where the target system allows it, in the reason or comment field of each change. The record then shows both facts truthfully: the agent made the change, at the request of a named person.

The Model Context Protocol (MCP), a common standard for connecting agents to tools, reaches the same conclusion from a security angle. Its security guidance forbids “token passthrough,” where a tool server accepts a token issued for something else and forwards it to a downstream system. One of the listed reasons is that downstream logs may show requests that appear to come from a different identity, which makes investigation and auditing harder. The rule is short: “MCP servers MUST NOT accept any tokens that were not explicitly issued for the MCP server.”¹¹

Permission Tiers for Agent Tools

OpenAI’s practical guide to building agents suggests giving each tool a low, medium, or high risk rating based on read or write access, reversibility, the account permissions it needs, and financial impact, and using that rating to decide when to pause for checks or hand off to a person.¹³ For GxP, a four-tier version works well.

Enforce Limits Outside the Model

An instruction in the system prompt telling the agent not to delete records is a request, not a control. Google’s published approach to agent security rests on three principles: “agents must have well-defined human controllers,” “their powers must be carefully limited,” and “their actions and planning must be observable.” It pairs reasoning-based defenses with traditional deterministic security controls, rather than relying on either alone.¹⁴ In validation terms, that means the limits you rely on must be built into code and configuration you can test:

• In the target system: the agent’s account has only the roles its tool inventory needs. If the agent tries a Tier 4 action, the system refuses it.
• In the tool layer: each tool checks its parameters against allowlists (sites, record types, status values), caps the number of records per call, and refuses anything outside them before the call reaches the target system.
• In the orchestration layer: limits on the number of tool calls per task, the number of retries, and total run time, so a confused agent stops rather than loops.

The MCP guidance adds a useful pattern for access scopes: start with a minimal set covering low-risk read operations, ask for more only when a privileged operation is attempted, and log each elevation with the scope requested and the scope granted.¹¹ Wildcard or all-access scopes appear on its list of common mistakes.¹¹

Keep Agent Accounts in Your Access Reviews

The draft Annex 11 expects recurrent reviews of user accounts and roles, with managers confirming continued access.² Agent accounts belong in the same review, with a named owner who confirms that the rights still match the current tool inventory. Agent accounts are easy to forget because no person logs in with them, which is exactly why they need an owner.

Logging Every Call: What the Record Has to Show

If you cannot reconstruct what the agent did, and why, you cannot defend any record it touched. Logging for an agent has two layers, and GxP teams often have only one of them.

Layer One: The Target System’s Audit Trail

Your validated systems already have audit trails. When the agent writes a record under its own identity, the target system captures who, what, and when, including old and new values. That is necessary, and it only works if the identity pattern from the previous section is right.

But notice how the draft Annex 11 frames audit trails. It asks for audit trail functionality that automatically logs “all manual user interactions,” and it asks systems to prompt the user for the reason when data is changed.² An agent’s write is not a manual user interaction, and an agent will not answer a reason-for-change prompt the way a person does unless you design it to. The target system’s audit trail will show that the agent account changed a field. It will not show what the agent was asked to do, what it read before deciding, or which person asked for the work.

Layer Two: The Agent Action Log

The second layer is a log the agent platform writes for every task and every tool call. 21 CFR 11.10(e) sets the baseline for any audit trail: secure, computer-generated, time-stamped, recording the date and time of entries and actions that create, modify, or delete electronic records, retained at least as long as the records themselves.⁵ For an agent, the content needs to go further than a standard system audit trail.

Standard Telemetry Will Not Capture This by Default

Engineering teams will likely instrument the agent with OpenTelemetry, the common open standard for traces, and its conventions for generative AI include an execute_tool span for each tool call. The tool name is required. But the tool call arguments and the tool call result are both marked Opt-In, and the conventions themselves are still at Development status.¹² In other words, a default setup records that the agent called a tool, not what it sent or what came back. For GxP, those are the two things you need most.

The Reason for Change, Written by an Agent

The draft Annex 11 asks for the reason a change was made to be captured.² For an agent, the honest reason is usually the task and the person who requested it. Design the write tools to pass a structured reason into the target system’s comment or reason field, such as the task ID, the requesting person, and a short description of the request. That keeps the target system record meaningful on its own, even when a reviewer never opens the agent log.

Sensitive Data in Logs

Tool arguments and results can contain personal data, particularly in clinical and pharmacovigilance systems. The OpenTelemetry conventions warn that tool attributes may contain sensitive information.¹² The answer is to control access to the log, not to drop the arguments. A log that leaves out what the agent sent cannot support an investigation.

Reviewing Agent Logs

The draft Annex 11 expects audit trail reviews to follow a documented procedure, be done by people not involved in the activity, and be targeted by risk rather than covering every entry.² For agent logs, useful targets include denied calls, retries and loops, scope elevation events, writes outside normal hours, sudden changes in call volume, tool calls to records outside the requested product or site, and approvals given within seconds of the request. These patterns point to either a misbehaving agent or a checkpoint that is not doing its job.

One more point: test the logging itself during validation. For each test case, check that the expected log entries exist, are complete, and match the target system audit trail. A log that was never tested is an assumption.

This article focuses on what validation needs from logging. For the full layered design of audit records for agents, see our earlier piece on audit trail architecture for agentic AI, listed under Further Reading.

Failure-Mode Testing: Proving How the Agent Behaves When Things Break

A test plan that only shows the agent completing its intended tasks tells you very little. The draft Annex 11 names the areas that deserve more testing focus, including access privileges, audit trails, error handling, handling of alarms and warnings, boundary and negative testing, and reports and interfaces.² For an agent that calls other systems, error handling, negative testing, and interfaces are most of the risk.

Use Scenario and Exploratory Testing, Not Only Scripts

FDA’s CSA guidance describes unscripted testing methods that suit agents well. Scenario testing is defined as a technique “based on exercising sequences of interactions between the test item and other systems,” with users counted as other systems. Error guessing derives test cases from knowledge of past failures or general failure modes. Exploratory testing lets a skilled tester design tests as they go, looking for hidden and unanticipated behavior.⁴ The guidance also notes that unscripted testing may be better suited than scripted testing to assure that software performs as intended, even for high process risk functions.⁴

That definition of scenario testing describes an agent almost exactly: a sequence of interactions between the agent and other systems. A sensible plan uses scripted tests for the controls that must never fail (permission refusals, approval enforcement, log completeness) and scenario and exploratory testing for the agent’s handling of realistic, messy work.

A Failure-Mode Catalog for Agents That Call Systems

Test for the Attack Classes Now Seen in Practice

Microsoft’s AI Red Team updated its taxonomy of failure modes in agentic AI systems in June 2026 after a year of red teaming, adding seven new failure modes. They include goal hijacking, where adversarial instructions redirect an agent’s objective while the task still looks legitimate; inter-agent trust escalation, where a compromised agent claims false identity or permissions to an orchestrator; session context contamination, where biasing data introduced early in a session shapes later steps without tripping any single check; and MCP or plugin abuse, including poisoned tool descriptions.¹⁰ The team recommends treating the new categories as mandatory test classes for any agent that touches production data, and testing them at the system level across complete task flows rather than one step at a time.¹⁰

For a GxP agent, the practical version is a set of test cases where the planted instruction arrives through the systems the agent reads in normal work: a free-text field in a deviation, a comment in a change request, a supplier document, or a tool description changed in the test environment. The pass condition is not that the agent ignores every attempt. It is that no attempt produces a write outside scope, because the permission and tool-layer controls block it even when the agent is fooled.

Run Critical Scenarios More Than Once

Because the same request can produce different paths, a single pass is weak evidence. The τ-bench authors proposed a pass^k measure, which counts a task as passed only if the agent succeeds on all k attempts, as a way to measure consistency rather than one-off success.⁶ CAISI found the same thing from the attack side: moving from one attempt to 25 raised average hijacking success from 57% to 80%.⁸

Build this into your acceptance criteria. Set the number of repeated runs for each critical scenario based on risk, and write down why. Treat prohibited outcomes differently from task quality: any prohibited action in any run is a failure, whatever the success rate of the other runs. Task quality measures, such as how often the draft is complete and correct, can carry a documented threshold.

Test Against Realistic Systems

Anthropic’s tool guidance recommends evaluating agents on tasks drawn from real-world use, with realistic data and services, and warns against overly simple sandbox environments.¹⁵ In GxP terms, that means a validation environment with configured instances of the target systems, realistic record volumes, and the kinds of messy entries real users create. Mocks are fine for development. They are not enough for validation evidence, because many of the failures that matter come from the real system’s behavior: its error messages, its field rules, its timeouts.

Human Checkpoints That Check Something

Almost every GxP agent design includes a human-in-the-loop somewhere. The question is whether that person is performing a real check or just clicking through.

What the Draft Annex 22 Asks of the Human

The draft Annex 22 does not treat human review as a free pass. Where a model gives input to a decision made by a person, and testing of the model has been reduced on that basis, the intended use should describe the operator’s responsibility, and the operator’s training and consistent performance should be monitored like any other manual process.¹ It also asks that records be kept of that review and notes that, depending on criticality and the level of testing, this may mean reviewing or testing every output according to a procedure.¹

That is the core point. If you reduce agent testing because a person reviews its work, the person’s review becomes a control you must specify, train, record, and monitor.

Where to Place Checkpoints

OpenAI’s guide names two triggers that usually warrant handing control to a person: exceeding failure thresholds, such as a limit on retries or actions, and high-risk actions, meaning actions that are sensitive, irreversible, or high stakes, which should require human oversight until confidence in the agent’s reliability grows.¹³ Mapped to the permission tiers above:

• Tier 1 reads: no checkpoint needed per call, but the output of the task is reviewed.
• Tier 2 drafts: the checkpoint is the normal review and approval workflow in the target system. The draft does nothing until a qualified person acts on it.
• Tier 3 reversible writes: a confirmation step before the call runs, showing the exact change.
• Tier 4 decisions: made by a person in the target system under their own identity. The agent does not hold the permission.
• Any tier, after repeated failures: the orchestration layer stops the run and routes it to a person.

Designing an Approval Screen That Cannot Be Misread

Microsoft’s red team found that approval steps themselves can be attacked or worked around. Its recommendations include breaking compound actions into their parts before presenting them for approval, summarizing agent-written descriptions independently so an agent cannot hide what an action does behind a harmless description, scaling approval requirements to how reversible an action is and how far its effects reach, invoking the approval step deterministically rather than leaving it to the agent, and watching for unusual patterns in approval requests.¹⁰

For a GxP approval screen, that translates into a few design rules:

• Show the target system, the record ID, the field, the current value, and the proposed value, generated from the actual tool call, not from the agent’s description of it.
• Present one change per approval, or list every change in a batch individually.
• Enforce the approval in the tool layer: a Tier 3 call without a valid approval reference is rejected.
• Record the approver’s identity, the time, and exactly what was displayed.
• If the approval is itself a GxP decision, treat it as one: it belongs in the target system under the person’s own identity, with an electronic signature where the process requires one.

Watch for Approvals Without Review

A person who approves hundreds of agent actions a week will, over time, approve some without reading them. You can measure this. Track time from request to approval, approval rates, and how often approvers change or reject what the agent proposed. An approval rate close to 100% with very short review times is a signal to investigate, retrain, or redesign the workflow. This monitoring is what the draft Annex 22 means by watching the operator’s consistent performance.

Staying Validated After Go-Live

An agent that calls other systems has an unusual weakness after go-live: its validated state depends on systems it does not own. A LIMS upgrade that renames a field, a new status value added to the QMS workflow, or an API version change can alter what the agent does, even though no one touched the agent.

Change Triggers Specific to Agents

The draft Annex 22 asks that a tested model, the system it runs in, and the process it supports be placed under change control before deployment, and that any change to the model, the system, or the process be evaluated to decide whether retesting is needed.¹ It also asks for configuration control with effective measures to detect unauthorized change.¹ For agents that call other systems, the list of changes that should trigger that evaluation is longer than usual:

• a new model or model version
• a change to the system instructions
• a change to any tool description
• a change to tool code or parameter limits
• a new tool added to the inventory
• a change to the agent account’s roles or scopes
• an upgrade, configuration change, or API change in any target system
• a change to the approval workflow or the log store

The last three are the ones most often missed, because they come from other teams. The fix is procedural. Record the agent as a dependent interface in each target system’s change control, so that a planned upgrade to the QMS or LIMS prompts an impact assessment for the agent. The draft Annex 11 already expects critical data transfer between systems to rely on validated interfaces.² The agent is one of those interfaces, and its owners need to hear about changes on the other side.

Microsoft’s red team also recommends keeping a software bill of materials for every deployed agent that covers plugins, MCP servers, prompt templates, and tool descriptions.¹⁰ That list maps directly onto the configuration specification from the scope section, and it gives change control something concrete to compare against.

A Regression Suite Built From Your Failure-Mode Tests

The failure-mode tests from validation should become a regression suite that runs on every change trigger. Automate what you can, particularly the permission refusals, approval enforcement checks, and planted-instruction tests. These are the tests most likely to catch a change that looks harmless.

Monitoring in Operation

The draft Annex 22 asks for regular monitoring of model performance and of whether inputs remain within the intended use.¹ For an agent, useful operational measures include task completion rate, the rate of denied tool calls, escalations to people, approver change and rejection rates, and requests that fall outside the documented task boundary. A rise in denied calls or out-of-scope requests usually means users are pushing the agent toward work it was not validated for. That is a signal to either expand scope through change control or tighten the refusal behavior.

Include the agent in periodic review as you would any validated system, and review the agent action log against the target system audit trails on a sampled basis to confirm the two still agree.

What Goes in the Validation Package

Pulling the pieces together, a validation package for an agent that calls other systems has seven parts. Most of them are familiar deliverables adapted to an agent. A few are new.

None of this requires a new validation methodology. It requires applying the one you have to a new kind of user, and being honest about which controls depend on the model and which are enforced by code you can test.

Conclusion

An AI agent that calls other systems is best treated as a new, non-human user of your validated systems, with a very wide range of possible behavior. You will never test every path it might take, and the research is clear that its own judgment is not reliable enough to be the main control. What you can validate is the boundary: a written scope that says what it is for and what it may not touch, a unique identity with permissions enforced by the target systems and the tool layer, a complete record of every call that QA can review, tested behavior when systems fail or data is hostile, and human checkpoints that are enforced in code and monitored in practice. Each of these draws directly on principles already in Annex 11, Part 11, and CSA. With draft Annex 22 still excluding language models from critical GMP applications, keeping agents in supporting roles with qualified people responsible for the outcome is both the regulatory reading and the practical one.

Sakara Digital works with pharma and biotech organizations that are moving AI agents from pilots into validated environments. If you are working out how to scope, test, and document an agent that touches your QMS, LIMS, or document systems, and want an independent perspective on where to start, we are happy to have that conversation.

For Further Reading