Hallucination Testing for GxP Document Drafting Tools: A Practical Acceptance Test

Why Published Hallucination Rates Cannot Answer Your Question

When a quality or IT leader first asks how often a drafting tool makes things up, the usual answer is a number from a paper or a vendor slide. There are plenty to choose from. The trouble is that each number measures a different thing, in a different unit, on a different task, and the spread is wide enough that picking one tells you more about the person choosing than about your tool.

Citation Fabrication Studies Measure One Narrow Behavior

The most quoted figures come from studies that asked a general chatbot to write with references and then checked those references. Bhattacharyya and colleagues had ChatGPT write medical papers and checked the 115 references it produced. They found 47% fabricated, 46% authentic but inaccurate, and 7% both authentic and accurate. An incorrect PubMed identifier was the most common error, appearing in 93% of the papers.¹ Chelli and colleagues asked GPT-3.5, GPT-4, and Bard to find the references for 11 published systematic reviews and compared the output against the real reference lists. Hallucination rates were 39.6% for GPT-3.5, 28.6% for GPT-4, and 91.4% for Bard.² A 2025 study by Linardon and colleagues had GPT-4o write six literature reviews in mental health and found that 35 of 176 citations (19.9%) were fabricated, and that 64 of the 141 real citations (45.4%) contained errors, most often a wrong or invalid DOI.³

These are useful warnings. They are not hallucination rates for a drafting tool. Each study used a general chatbot working from its own memory, with no source documents supplied. A GxP drafting tool is supposed to work from a defined source package, which is a different task with a different failure pattern. The studies also define “hallucinated” differently. Chelli and colleagues counted a reference as hallucinated if any two of title, first author, and year of publication were wrong.² Linardon and colleagues counted a citation as real only if a publication closely matched both its title and its authors, and logged other errors separately.³ A reference with a correct title and a wrong year would count as accurate in one design and erroneous in another.

Linardon’s study adds a finding that matters for pharma. Fabrication rates varied by topic: 6% for major depressive disorder, against 28% for binge eating disorder and 29% for body dysmorphic disorder, which have smaller research literatures.³ Rare disease, early-stage modalities, and specialized CMC topics are exactly where the literature is thin. A test set built only from well-studied areas will understate the rate in the areas where a biotech does most of its writing.

Grounded Tasks Have Lower Rates, Not Zero Rates

Studies where the model works from supplied documents show lower rates, which is the reason drafting tools are built on retrieval in the first place. Asgari and colleagues measured GPT-4 drafting clinical notes from consultation transcripts. Across 12,999 sentences, they found a hallucination rate of 1.47% and an omission rate of 3.45%, and 44% of the hallucinations were classed as major.⁵ All of the authors were employees of the company that makes the tool under study, which is worth keeping in mind, though the method is published in enough detail to reuse.

The Vectara hallucination leaderboard asks each listed model to summarize the same set of more than 7,700 articles using only the information in each article, and then scores factual consistency with its own detection model. At its October 22, 2026 update, the listed rates ran from 1.8% to 24.2%.⁶ The page states plainly that it does not evaluate the quality of the summaries, only their factual consistency, and that it does not claim to have solved hallucination detection.⁶ A leaderboard score is a reasonable input to vendor selection. It is not evidence about your use, because the documents, the instructions, and the scoring model all differ from yours.

Retrieval also does not make a tool safe by itself. Magesh and colleagues ran a preregistered evaluation of commercial legal research tools that retrieve and cite real case law. Each tool hallucinated more than 17% of the time.⁴ Their definition is worth borrowing: a response counts as hallucinated if it makes a false statement or falsely asserts that a source supports a statement.⁴ The second half of that definition covers the failure that general benchmarks tend to miss: a statement that is true, attached to a source that does not support it.

What the Research Does Tell You

Read together, the studies support four conclusions that should shape your test. First, the unit of counting changes the number by an order of magnitude, so your acceptance criteria must name the unit. Second, the rate depends on topic and on how much the model has to supply from memory. Third, longer inputs make things worse: Tang and colleagues found that giving ChatGPT a longer source text lowered how well it picked out key information and raised the likelihood of factual inconsistency errors in medical evidence summaries.⁷ Fourth, models tend to accept the premise they are handed. Dahl and colleagues found that models often failed to correct a user’s incorrect legal assumption when asked a question built on it.⁸ A drafting instruction such as “describe the improvement in the secondary endpoint” carries exactly that kind of premise.

Three Kinds of Hallucination in Regulated Writing

The research literature has several taxonomies. Huang and colleagues, in a widely cited survey, split hallucination into factuality hallucination (output that conflicts with verifiable real-world facts) and faithfulness hallucination (output that departs from the user’s instructions or supplied context, or contradicts itself).⁹ Maynez and colleagues, working on summarization, separated intrinsic hallucinations, which misrepresent information that is in the source, from extrinsic hallucinations, which add information the source does not contain.¹⁰ The RAGTruth corpus uses four labels for retrieval-based output: evident conflict, subtle conflict, evident introduction of baseless information, and subtle introduction of baseless information.¹¹

These are sound research categories, but they do not map cleanly onto the decisions a GxP reviewer makes. For a drafting tool that works from a source package, a three-part definition is easier to apply and easier to defend in an inspection, because each type points to a different check and a different fix.

Factual Hallucination

This is the type most people mean by the word. In a GxP drafting context, it shows up as the wrong number of subjects randomized, an adverse event count that does not match the table, a batch number that does not exist, a stability timepoint the study never had, or a reference to a publication that cannot be found. It also includes reversals: stating that a difference was statistically significant when the analysis says it was not. Tang and colleagues described this as a misinterpretation error in medical evidence summaries, where the conclusion drawn in the summary contradicts the results in the source.⁷

Invented citations deserve their own line in the rubric even when the tool is grounded. If the tool is allowed to add references beyond the package (for a background section or a literature summary, for example), the citation studies above apply directly. If it is not allowed to, any reference outside the package is a defect whether or not it exists.

Structural Hallucination

Regulated documents follow set structures. ICH E3 lays out the sections of a clinical study report, from the synopsis through the efficacy and safety evaluations to the tables, listings, and appendices.¹³ Deviation reports, validation summaries, and annual product reviews follow company templates. A drafting tool given a template tends to fill every slot. When a section does not apply, a careful writer states that and explains why. A tool may instead produce plausible content for it: an interim analysis section for a study that had none, a subgroup table for subgroups that were never analyzed.

Structural hallucination also covers tables. When a tool rebuilds a summary table in prose or in a new table, it can add a row, drop a row, merge two categories, or produce a value for a cell that was blank in the source. In a challenge organized by Pfizer to generate safety table summaries for clinical study reports, Landman and colleagues found that participating solutions differed in performance, particularly in factual accuracy, and that one team’s fine-tuned model improved on the automated metrics while still producing hallucinations and erroneous summaries.¹² Good metric scores and correct tables are not the same thing.

Contextual Hallucination

This is the type that does the most damage in pharma writing, because every individual fact in the sentence checks out. A drafting tool working across a program’s documents can take the response rate from the Phase 2 study and place it in the Phase 3 report. It can report the safety population count where the efficacy population belongs. It can quote the inclusion criteria from protocol version 3 when version 4 was in force for the study period. It can cite a real, relevant paper for a claim the paper does not make.

Magesh and colleagues call this last case “misgrounded”: the source is cited but does not support the claim, or the source is inapplicable.⁴ Asgari and colleagues also kept a separate “contextual” category in their clinical note taxonomy, alongside fabrication, negation, and causality errors; contextual errors were 17% of the hallucinations they found, and negation errors, which they flagged as the most concerning, were 30%.⁵ Most public benchmarks give each model one document at a time, so they cannot provoke the wrong-study error at all. Your test has to be built so that it can.

Why Omission Belongs in the Same Test

Omission is not a hallucination, but a tool tuned to avoid saying anything unsupported can reach that goal by saying less. Asgari and colleagues found more omissions than hallucinations, 3.45% of source sentences against 1.47% of output sentences.⁵ FActScore, a widely used method that breaks output into atomic facts and checks each one against a knowledge source, measures precision but not recall, which its authors note directly.¹⁵ A test that only counts unsupported claims would rate a tool that leaves out the serious adverse events as perfect. Score omissions alongside hallucinations, against a list of facts the draft must contain.

Building the Test Set From Real Source Packages

The test set is the part of the work that decides whether the result means anything. For a drafting tool, the unit of the test set is a source package: the complete set of inputs the tool would receive for one real drafting task, plus the instruction and template it would be given, plus a reference record of what a correct draft must and must not contain.

What Goes in a Source Package

Take the packages from completed, closed work, so that the correct answer is known and settled. For a clinical study report section, a package might hold the protocol and its amendments, the statistical analysis plan, the relevant tables, listings, and figures, and the instruction the author would give. For a deviation report, it holds the event record, batch documentation extracts, investigation notes, and the CAPA record. For a validation summary, it holds the test protocols, executed results, and the deviation log.

Each package needs a reference record, approved by a subject matter expert, that lists the key facts the draft must state (with their exact source locations), the facts it must not state, the sections that must be marked not applicable, and the correct handling of each challenge case built into the package. The draft EU GMP Annex 22 asks that test labels be verified through a process that gives a very high degree of correctness, which may include independent verification by more than one expert.¹⁶ Annex 22 is a consultation draft, and its scope excludes generative AI and large language models from critical GMP uses, but its test data principles transfer well to a drafting tool.

Stratify by Document Type and Difficulty

Annex 22 asks that test data represent the full sample space of the intended use, be stratified to include all subgroups, and reflect common and rare variations, with the selection rationale documented.¹⁶ For a drafting tool, the strata that matter are the document types in scope, the therapeutic or product areas, the size of the source package, and the difficulty of the content. Include long packages on purpose, since Tang and colleagues found more factual inconsistency with longer inputs.⁷ Include thin-literature topics on purpose, since Linardon and colleagues found more fabrication where the research base was smaller.³

Build Challenge Cases Into the Packages

A test set of ordinary packages will tell you the baseline rate. It will not tell you how the tool behaves at the points where hallucination is most likely. For that, you need challenge cases: packages, or changes to packages, built so that one specific error is possible and the correct behavior is known in advance. Each challenge case targets one of the three types.

Two of these cases deserve a note. The removed-value and citation-request cases test whether the tool will say “not found” when it should. Kalai and colleagues argue that language models hallucinate in part because training and evaluation reward guessing over acknowledging uncertainty.¹⁴ If your rubric scores a “not found” answer as an incomplete draft and a confident guess as a complete one, your test will reward the behavior you are trying to prevent. Score a correct “not found” as correct.

The false-premise case follows from Dahl and colleagues’ finding that models often fail to correct an incorrect assumption in the question.⁸ In drafting work, the premise arrives in the author’s instruction, and authors write instructions based on what they expect the data to show.

Keep the Test Set Independent and Real

Annex 22 asks for controls that keep test data out of model development, training, and validation, and it says that generating test data or labels with generative AI is not recommended and should be fully justified if done.¹⁶ For a drafting tool, the practical meaning is simple. The people who write and tune the prompts must not see the test packages. Synthetic source packages invented by a model are a poor substitute for real ones, because the model that wrote them shapes what errors are possible. Real packages from completed work, with challenge cases made by editing real documents under control, are the better choice. Keep a copy of each package exactly as tested, with a checksum, so the test can be repeated.

A Scoring Rubric by Type and Severity

A pass or fail grade on a whole draft hides almost everything a quality unit needs to know. The rubric below scores the draft claim by claim, records the type of each error, rates its severity, and keeps omissions in a separate count. Our earlier article on evaluating model output quality when there is no ground truth covers rubric writing and reviewer agreement in general terms. The rubric here is narrower, because a source package gives you a ground truth to check against.

Choose the Counting Unit

Use the checkable claim as the main unit. A claim is one statement that can be checked against the source on its own: a value, a date, a count, a direction of effect, an attribution. FActScore uses a similar unit, the atomic fact, defined as a short sentence carrying one piece of information.¹⁵ Two adjustments make it fit regulated writing. FActScore gives every fact equal weight, which its authors list as an assumption.¹⁵ Your rubric should not, because a wrong serious adverse event count matters more than a wrong date of a routine meeting. And tables need their own unit: score each cell in a table the tool produces as a claim.

Use the reference as the unit for citations and the section as the unit for structure. That gives four counts per draft: claims checked, references checked, sections checked, and required facts checked for omission. Report each separately. Combining them into one rate repeats the problem the published studies have.

Classify Every Claim

Each claim gets one of five outcomes:

• Supported: the claim matches the source at the location the reviewer can point to.
• Contradicted: the source says something different. Factual hallucination.
• Unsupported: nothing in the package supports or contradicts it. Factual hallucination if the tool was limited to the package; a judgment call otherwise.
• Misattributed: the claim is true somewhere in the package or program, but not for the study, population, timepoint, version, or reference it is attached to. Contextual hallucination.
• Correctly flagged: the tool said the information was missing or in conflict, and it was. Scored as correct.

For every non-supported claim, the reviewer records the exact source location that shows the error. That record is what makes the result reviewable by a second person and what lets you trace each error to its cause later.

Rate Severity by Consequence

Severity should reflect what the error could do if it reached the final document, not how obvious it was. The scale below has three levels for errors and treats omissions on the same scale.

Write examples like these for each document type in scope, approved by the subject matter expert before scoring starts. Asgari and colleagues used a similar two-level split, defining major errors as those that could affect diagnosis or patient management if not corrected.⁵ The definitions only work if reviewers apply them the same way, so test them on a few packages first and settle disagreements by revising the examples, not by discussion at the time of scoring.

Split the Work Between Scripts and People

Some checks are mechanical and should be scripted. Comparing every value in a generated table with the source table is a matching problem. Checking that each reference exists can be done against PubMed, Crossref, or your literature database. Checking the section list against the template is a comparison of headings. Scripted checks are fast, repeatable, and do not tire.

Contextual checks need people who know the program. Only a reviewer who knows which study, population, and version the sentence should draw on can tell whether a correct-looking value is misattributed. Automated faithfulness scores and model-based judges can help sort which drafts get human attention first, but Tang and colleagues found that common automatic summary metrics did not strongly correlate with human judgments of quality,⁷ and a model-based judge has its own error rate that must be measured before you rely on it. For an acceptance test, use automated scoring to add to human scoring of the challenge cases and the critical claims, never to replace it.

Acceptance Criteria Tied to Use Risk

There is no regulatory number for an acceptable hallucination rate in a drafted document, and the two most relevant regulatory drafts both stop short of this use. FDA’s January 2025 draft guidance on AI to support regulatory decision-making for drugs and biologics says it does not address AI used for operational efficiencies, and it lists drafting or writing a regulatory submission as an example, where that use does not affect patient safety, drug quality, or the reliability of study results.¹⁷ The draft Annex 22 excludes large language models from critical GMP applications and says that, in non-critical uses, qualified people should be responsible for confirming the output is suitable, with its principles to be considered where they apply.¹⁶

So the acceptance criteria are yours to set and defend. Both drafts still give you a sound structure. FDA’s draft frames model risk as a combination of model influence (how much the output contributes to the decision compared with other evidence) and decision consequence (how serious an adverse outcome from a wrong decision would be).¹⁷ Annex 22 asks that acceptance criteria be set by a process subject matter expert and approved before acceptance testing starts, and that a model’s acceptance criteria be at least as high as the performance of the process it replaces.¹⁶

Define Use Tiers

For drafting tools, three tiers cover most uses:

• Tier 1, working drafts. Output that helps an author start but does not become part of a GxP record: a first outline, an internal meeting summary, a background note. The author rewrites it.
• Tier 2, drafts of GxP records. Output that becomes a controlled record after qualified review and approval: a deviation report narrative, a validation summary, an SOP revision, a periodic review.
• Tier 3, decision-bearing and regulator-facing content. Output that informs a product quality or patient safety decision, or goes to a regulator: clinical study report sections, safety narratives, investigator’s brochure updates, annual product review conclusions, Module 2 summaries.

Write the tier into the intended use. A tool qualified for Tier 2 deviation narratives is not qualified for Tier 3 safety narratives because the drafting looks similar.

Set the Criteria by Tier and Severity

Express each criterion as an upper confidence bound on a rate per counting unit, not as an observed rate. An observed rate of zero in 50 claims tells you very little; an upper bound tells the reader how rare the error has been shown to be. The table shows one way to structure the criteria. The numbers are examples to show the structure, not recommended values. Your subject matter experts must set and approve their own before the test runs.

Measure the Human Baseline

The “at least as good as the process it replaces” principle in Annex 22 requires knowing how the current process performs.¹⁶ Most organizations do not know their writers’ error rate on a given document type, because errors are corrected in review and not counted. The cleanest way to get a baseline is to have experienced writers draft a subset of the same test packages under normal conditions and score their drafts with the same rubric, without telling the scorers which drafts came from people. Asgari and colleagues used the same kind of comparison, reporting that by refining prompts and workflows they reduced major errors below previously reported human note-taking rates.⁵

A baseline also protects you from a criterion that cannot be met. If skilled writers make one major error per few hundred claims in a first draft, a criterion that demands the tool make none is not a quality standard. It is a reason to never use the tool. The point of the baseline is to compare like with like: first draft against first draft, before review.

How Many Samples Are Enough

The question of sample size for an acceptance test is a question about rare events. You are trying to show that a critical error is uncommon, and the less common you need it to be, the more claims you must check without finding one.

The Zero-Failure Math

If you check n independent claims and find no errors of a given kind, the one-sided 95% upper bound on the true rate is close to 3 divided by n. Hanley and Lippman-Hand set out this “rule of three” for interpreting zero numerators in a 1983 JAMA paper.¹⁸ The exact binomial figures below show how many claims you need to check for a target upper bound, depending on how many errors you allow yourself to find.

Two things follow. First, a test of ten drafts with no critical errors found proves very little, even if each draft looked clean. Ten drafts might hold 400 checkable claims, and zero errors in 400 independent claims supports an upper bound of about 0.75%, which may not meet a Tier 3 criterion. Second, plan the sample size before the test and decide in advance what happens if one error is found. If the plan assumes zero and one turns up, the bound moves a long way.

Claims in the Same Draft Are Not Independent

The table assumes independent claims. Claims in one draft are not independent. If the tool misreads which study it is drafting for, every value in the draft may be wrong at once. Statisticians handle this with a design effect: the ratio by which clustering inflates the variance compared with independent sampling. A common approximation is 1 + (m − 1) × ICC, where m is the number of claims per draft and ICC is the correlation between claims in the same draft. With 60 claims per draft and a modest correlation of 0.05, the design effect is about 4, which means 600 claims from 10 drafts carry roughly the information of 150 independent claims.

You will not know the correlation in advance. The practical response is to spread the claims over as many packages as you can, rather than checking every claim in a few long drafts. Fifty packages with 30 checked claims each is a much stronger test than ten packages with 150 claims each, even though the total is the same. If the critical errors you find cluster in a few packages, report that, and treat the package rather than the claim as the unit for that criterion.

Strata Each Need Their Own Evidence

Annex 22 asks that the test set and each of its subgroups be large enough to calculate the test metrics with adequate statistical confidence.¹⁶ If the intended use covers four document types and you want to claim the critical rate is below 0.5% for each, you need roughly 600 clean claims in each, not 600 in total. This is the strongest argument for keeping the first intended use narrow. A tool qualified for one document type, with strong evidence, is more useful than a tool qualified for six on thin evidence.

Repeat Runs Measure Variation

Drafting tools do not always produce the same output from the same input. The Vectara leaderboard used a temperature of zero where the model allowed it, to reduce run-to-run variation,⁶ but many production tools do not offer that setting or do not guarantee identical output with it. Run a subset of packages two or three times. If the error pattern changes between runs of the same package, report the variation, and count each run’s claims as clustered within the package, not as new independent evidence.

Re-Running the Test After Model Changes

An acceptance test describes one configuration: one model version, one set of prompts and templates, one retrieval setup. The draft Annex 22 asks that a tested model and the process around it be put under change control, and that any change to the model, the system, or the process be evaluated to decide whether retesting is needed, with any decision not to retest fully justified.¹⁶ Our article on change control for prompt and model updates covers the change classes, the sealed regression set, and vendor update handling in detail. Four points are specific to hallucination testing.

Re-Run Every Challenge Case

Ordinary packages can be sampled for a re-test. Challenge cases should not be. They are the part of the test aimed at the errors that matter most, and a model update can change behavior on a specific challenge without moving the average. Run the full challenge catalog on every model change that the impact assessment says needs retesting.

Compare by Type, Not Only in Total

A new model version can lower one kind of hallucination and raise another. A version that fabricates fewer citations may be more willing to fill a removed value, or more likely to pull a value from a sibling study because its retrieval behaves differently. Report the re-test by type and severity, set the comparison criteria before the run, and look at which individual claims changed from correct to incorrect, not just at the totals.

Keep the Scoring Fixed

Use the same rubric, the same severity examples, and where possible the same reviewers. If the rubric changes between runs, you cannot tell a change in the tool from a change in the scoring. When the rubric does need revising, re-score the previous run’s output with the new rubric so both runs are measured the same way.

Add Real Incidents to the Catalog

Every hallucination found in production review, once its cause is understood, should become a new challenge case in the next version of the test set. Over time, the catalog shifts from cases someone imagined to cases the tool has failed on. Xu and colleagues argue, on formal grounds, that hallucination cannot be eliminated entirely in language models used as general problem solvers.¹⁹ Whatever one makes of the theory, the operating assumption should be the same: the tool will keep producing some errors, and the test set should grow with them.

Running the First Acceptance Test, Step by Step

The sequence below brings the pieces together. It assumes the intended use and use tier are written and approved, and that the tool is configured and frozen for the test.

What the Report Should Show

The acceptance test report should let a reader who was not involved repeat the test and reach the same conclusion. It should include the approved plan; the package list with checksums and selection rationale; the reference records; the tool configuration tested; the raw outputs; the claim-level scoring with source locations for each error; the rates and bounds by type, severity, and stratum; the human baseline comparison; the investigation of each critical and major error; and a clear statement of what the tool is and is not qualified for. Annex 22 asks that test documentation be kept along with the intended use description, the characterization of the test data, and the test data itself.¹⁶

The last element, the statement of scope, is the one most often left vague. “Qualified for drafting” is not a scope. “Qualified to produce first drafts of deviation report narratives for packaging deviations at the two sites in the test set, for review and approval under the deviation SOP” is. If the tool will later be used for another document type or site, that is a change, and it gets its own test.

Conclusion

Hallucination testing for a GxP drafting tool comes down to a few disciplined choices. Define the errors in terms a reviewer can apply: factual, structural, and contextual, with omissions scored alongside. Test on real source packages with challenge cases built to provoke each type, especially the wrong-study and wrong-version errors that public benchmarks cannot see. Count claims, not documents, rate each error by what it could do, and set criteria as upper bounds that tighten as the output moves closer to a regulatory or quality decision. Size the test from the math of rare events, spread it across many packages, and re-run the challenge cases every time the model or configuration changes. None of this needs new science. It needs the same care quality teams already bring to method validation, applied to a tool whose errors are easy to read past.

Sakara Digital works with pharma and biotech organizations building the evidence behind AI tools used in regulated work, from intended use statements through acceptance testing and change control. If you are preparing to qualify a drafting tool and want an independent view on how to design a test that will stand up to review, we are happy to have that conversation.

For Further Reading