Evaluating Model Output Quality When There Is No Ground Truth

The Tasks Where No Right Answer Exists

The distinction that matters is not between traditional software and AI. It is between tasks with a determinable correct output and tasks without one. A great deal of confusion in life sciences AI programs comes from applying one mental model to both.

Consider two systems that both use a large language model. The first extracts the batch number, product code, and event date from a scanned deviation form. The second writes the summary narrative for that deviation. The first has ground truth. Someone can read the form, record the correct batch number, and compare. Accuracy, precision, and recall are all computable, and a test set of a few hundred documents supports a defensible statement about performance. The second has no ground truth at all. Two experienced quality investigators asked to summarize the same deviation will produce different narratives, both acceptable, differing in emphasis, order, and what they chose to leave out.

The second category is where most of the current interest in pharma and biotech is concentrated:

• Regulatory document drafting. Non-clinical overviews, clinical summary sections, responses to health authority questions, protocol synopses, and investigator brochure updates. Multiple compliant drafts exist for any given input.
• Deviation and investigation summarization. Condensing an investigation record into a narrative for a quality review board or an annual product quality review.
• Literature review and evidence synthesis. Screening, extracting, and summarizing across a search result set for safety signal work, medical affairs, or a pharmacovigilance periodic report.
• Query response. Medical information responses, internal question answering over standard operating procedures, and support for regulatory intelligence requests.

In every one of these, the reference answer either does not exist or exists in a form that is itself only one acceptable option among many. The absence of a reference is not a reason to skip evaluation. It changes which evaluation methods are available and it changes what the evidence looks like when written down.

Why “have a person check it” is not an evaluation strategy

Human review is the foundation of every method in this article. What fails is undefined human review. When a reviewer is handed an output and asked whether it is good, the answer carries no information about the system, because it mixes the reviewer’s standard, mood, familiarity with the input, and tolerance for a particular writing style into a single number that no one else can reproduce.

The language technology field documented this problem carefully. A survey of 165 papers on natural language generation found that generated text was evaluated along 204 distinct quality dimensions, which the authors mapped down to 71 separate criteria. What different papers called “fluency” broke into 15 different criteria depending on how the term was defined and used. Of 478 quality evaluation questions studied, more than half did not define the criterion being evaluated, 65 percent did not report the exact question given to evaluators, and 20 percent did not even name the criterion.⁸ That is the research literature, where methods sections are peer reviewed. Internal pilot evaluations in industry are rarely more disciplined.

What the regulators actually say, and what they do not

It is worth being precise here, because overstatement in this area does real damage to a program. No regulator has published a binding requirement telling you which evaluation method to use for a generative output without ground truth, or what agreement statistic to hit.

The European Medicines Agency’s reflection paper on the use of artificial intelligence in the medicinal product lifecycle sets out current thinking rather than binding requirements, and takes a risk-based position on development, deployment, and performance monitoring.¹⁹ EudraLex Annex 11 on computerized systems, which does carry regulatory weight in the EU, requires validation appropriate to risk and requires that testing evidence be documented and reviewed, without prescribing a method for language output.²⁰ The FDA’s draft guidance on artificial intelligence to support regulatory decision-making remains a draft and is addressed in a companion article in this series.

So the honest framing is this. Nobody is going to hand you an acceptance threshold. What inspectors and assessors consistently do expect, across every guidance in this space, is that you defined intended use, assessed risk, decided in advance what acceptable performance meant, tested against that decision, documented the result including the failures, and set up a way to notice when performance changes. Those expectations are method-agnostic. They are also the reason the documentation section of this article matters as much as the statistics.

Rubric Design: Criteria That Produce Agreement Instead of Opinion

A rubric is the instrument that turns a reviewer’s reaction into data. A bad rubric produces numbers that look like data and behave like opinion. The difference is almost entirely in how the criteria are written.

Decompose quality into attributes a reader can observe

The single most useful move is to stop asking about quality and start asking about specific, separately observable attributes. Clinical summarization research has converged on a small set that transfers well to regulated document work: completeness, correctness, and conciseness were the three attributes used in a physician reader study across four clinical summarization tasks.¹³ For regulated drafting and investigation summarization, a workable attribute set usually looks like this:

Note what is missing: no attribute called “quality”, “usefulness”, or “readiness”. Those are conclusions drawn from the attributes, not things a reviewer scores directly.

Clinical documentation has existing validated instruments worth borrowing from rather than reinventing. A review of evaluation practice for large language models in medical summarization names three pre-existing rubrics that teams have adapted: SaferDx, a retrospective survey focused on diagnostic error and missed opportunities; PDQI-9, which evaluates physician notes across nine criteria with emphasis on documentation organization; and Revised-IDEA, a four-item tool for assessing clinical reasoning documentation. The same review points out the problem with all of them in this context: they were designed for clinician-authored notes, not model outputs, and many evaluation frameworks in the published literature lack enough detail about how they were created to assess their validity, with inter-rater reliability not always reported.¹⁴

Prefer error taxonomies to Likert scales

A five-point scale invites a reviewer to average their reaction. An error taxonomy asks them to find and classify specific problems, which is a task with a much higher natural agreement rate.

In practice this means replacing “rate the factual accuracy of this summary from 1 to 5” with “mark every statement in this summary that is not supported by the source document, and classify each as: contradicted by the source, not present in the source, or overstated relative to the source.” Two reviewers can disagree about whether a summary deserves a 3 or a 4. They agree far more often about whether sentence six is contradicted by the source.

The error taxonomy also produces something a Likert scale never does: a defect record you can act on. A count of unsupported statements per thousand words, broken down by class, tells an engineering team what to fix. A mean score of 3.7 tells them nothing.

Write the reviewer instructions as a procedure, not a preamble

The instruction document a reviewer receives is the most underestimated artifact in the entire evaluation. It should specify the order in which they read the source and the output, whether they may consult other systems, how long they should spend, what to do when the output is ambiguous, and what to do when they believe the source material itself is wrong. Each of those choices changes the scores, and if the choice is not written down, different reviewers make it differently and the variation shows up as noise you will misread as model behavior.

The survey work on inter-coder agreement in computational linguistics makes the underlying point plainly: agreement coefficients measure the reliability of an annotation scheme and its guidelines, not just the diligence of the people applying it.⁹ Low agreement is usually a defect in the rubric before it is a defect in the reviewers.

Measuring Your Reviewers Before You Trust Them

This is the step almost every internal evaluation skips, and it is the step that decides whether anything else in the file means anything. If two qualified reviewers applying your rubric to the same output do not agree with each other, then the scores you collected describe your reviewers, not your model.

The coefficients, and which one to use

Percent agreement is not sufficient, because raters agree by chance at a rate that depends on how the categories are distributed. If 95 percent of outputs are acceptable, two reviewers who both mark everything acceptable will show 95 percent agreement while carrying no information at all. The chance-corrected coefficients exist to handle this.

For most regulated evaluation work, Krippendorff’s alpha is the better default. It does not require complete data, it accommodates any number of raters, and it extends across measurement levels, so the same coefficient covers a binary acceptable-or-not judgment and an ordinal severity scale.¹² Where the design is a clean two-rater categorical comparison, Cohen’s kappa is fine and is more familiar to reviewers.

The thresholds, and why the common ones are too generous

Cohen’s original interpretation bands are widely quoted: values at or below zero indicate no agreement, 0.01 to 0.20 none to slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, and 0.81 to 1.00 almost perfect. The methodological literature in health research argues directly that this is too lenient for health-related work, because it implies a value as low as 0.41 might be acceptable.¹⁰

Krippendorff’s own conventions are stricter and better suited to a regulated setting: rely on data at alpha of 0.800 or above, draw only tentative conclusions in the 0.667 to 0.800 band, and discard data below 0.667.¹² For a GxP evaluation, adopting the stricter convention is defensible and easy to justify. For a safety-relevant attribute, arguing for something higher still is reasonable.

What to do when agreement is low

Low agreement is diagnostic information, not a failure to hide. The response sequence that works:

There is a harder case worth naming. Sometimes agreement stays low after the rubric is clean and the panel is calibrated. That result is telling you the task genuinely has no stable standard among qualified experts. That is a legitimate finding, and the right response is to narrow the intended use rather than to keep adjusting the measurement until the number improves.

Paired Comparison and Preference Testing

Absolute scoring asks a reviewer to place an output on a scale. Paired comparison asks a much easier question: shown two outputs for the same input, which is better on a stated criterion? People are considerably more consistent at relative judgments than absolute ones, and the method removes the problem of scale drift across a long review session.

How the method works and what it produces

The reviewer sees two outputs, in randomized order, with the systems that produced them concealed. They pick one, or record a tie. Repeated across many items and many pairs, these votes are converted into a ranking. The public large-scale example of this design is Chatbot Arena, which collects crowdsourced pairwise votes and fits a Bradley-Terry model, a statistical model of paired comparison dating to 1952, to estimate relative strength from the vote record.²

For an internal evaluation the same structure works at much smaller scale. Useful comparisons include:

• Model output against the current human-authored baseline. The most relevant comparison for a document drafting use case, because the real decision is whether the model output is at least as good as what the process produces today.
• Candidate configuration against candidate configuration. Two prompt designs, two retrieval strategies, two model versions.
• Post-change against pre-change. After a model or configuration update, whether reviewers prefer the new outputs is a direct and interpretable signal.

The physician reader study in the clinical summarization work referenced earlier used this structure. Ten physicians compared adapted model summaries against summaries written by medical experts across four tasks. In a majority of cases the best-adapted model output was found either non-inferior, at 45 percent, or preferred, at 36 percent, with human summaries preferred in 19 percent.¹³ That result is only interpretable because the comparison was paired, blinded, and scored against defined attributes.

The limits you have to design around

Design controls that address these: randomize presentation order for every pair and record the order, blind the system identity, allow an explicit tie so reviewers do not force a distinction that is not there, require the reviewer to name the criterion that drove the choice, and run the absolute acceptability rubric on the same items so you have both signals from one review pass.

Expert Panels and the Adjudication of Disagreement

Clinical development solved a version of this problem decades ago. When an endpoint is subjective or requires the application of a complex definition, trials use a blinded, independent adjudication committee rather than relying on a single assessor. The FDA’s draft guidance on data monitoring committees is explicit that a data monitoring committee should not adjudicate endpoints, because it may hold unblinded comparative data, and that such adjudication belongs to a separate committee blinded to the assigned intervention.²¹ The structural insight transfers directly to model output evaluation.

The transferable pattern

Treat the disagreement rate as a primary output

Most teams report the final adjudicated scores and discard the record of how much adjudication was needed. That record is one of the more useful things the evaluation produces. A rising adjudication rate on a stable rubric with a stable panel indicates that the model’s output has moved into territory where experts no longer agree on acceptability, which is an early warning worth acting on. Track it as a metric with its own trend, not as an operational statistic about how long the review took.

Reference-Free Automatic Metrics and Why Most Are Weak

Automatic metrics are attractive because they scale. The honest position is that for open-ended generation in a regulated setting, most of them are too weak to carry a release decision, and a few narrow ones are genuinely useful.

The reference-based metrics do not apply, and would be weak anyway

ROUGE, BLEU, and their relatives compare a candidate against one or more reference texts. Without ground truth there is no reference to compare against, so they are unavailable by construction. Even where a reference exists, the correlation with human judgment is poor for anything requiring creativity or synthesis. Conventional reference-based metrics have relatively low correlation with human judgment, particularly for tasks needing creativity and diversity.³ A comparative study of summarization evaluation methods reported correlation between BERTScore, ROUGE-2, and human evaluation in the range of roughly 0.2 to 0.4, with weak correlation across measured dimensions.¹⁸ Earlier critical work on summarization evaluation reached the same conclusion from a different direction: existing automatic metrics do not adequately reflect the differences between model-written and human-written summaries.¹⁷

Embedding-based metrics such as BERTScore improve on surface overlap by comparing meaning rather than words, but they inherit the same problem in specialized domains, where correlation with expert judgment weakens further. A scoping review of clinical text summarization with large language models is worth reading on this point, as is the broader review of evaluation practice in medical summarization, which is direct about automated metrics failing for abstractive clinical summarization because surface matching does not capture clinical reasoning or detect fabricated content.¹⁴¹⁵

The narrow automatic checks that do earn their place

What works automatically is not a quality score. It is a set of deterministic checks that catch specific, well-defined defect classes with near-perfect reliability. These belong in the evaluation and in production monitoring, and they reduce the volume of material human reviewers have to process.

The pattern is that automatic checks handle the classes of defect where a correct answer is definable even though the output as a whole has no correct form. A citation either resolves or it does not. That is ground truth inside a task without ground truth, and it is worth mining thoroughly before anyone concludes that automation has nothing to offer here.

LLM-as-Judge: Real Uses and Documented Failure Modes

Using a language model to score another model’s output has become the default suggestion whenever human review looks expensive. It is a real method with a real research base and real, measured limitations. Both halves of that sentence matter.

What the evidence supports

The foundational work on this approach reported that strong model judges can match both controlled and crowdsourced human preferences at over 80 percent agreement, which is the same level of agreement observed between humans.¹ A separate framework for using a model with chain-of-thought prompting and a form-filling structure reported a Spearman correlation of 0.514 with human judgment on summarization, described as outperforming previous methods by a large margin.³ A 0.514 correlation is a genuine improvement over the 0.2 to 0.4 range typical of surface metrics. It is also nowhere near the agreement you would demand from a human reviewer panel before trusting its scores.

The picture gets less comfortable in specialist domains. A study of model judges against subject matter experts in dietetics and mental health found agreement on overall preference of 68 percent in dietetics and 64 percent in mental health, with agreement varying considerably on domain-specific aspects. The authors concluded that models alone may not provide the depth of understanding required for complex, knowledge-specific tasks.⁶ Pharmaceutical quality and regulatory work is a complex, knowledge-specific task by any reasonable reading of that finding.

The documented failure modes

A survey of the field organizes the known biases systematically, and an automated bias quantification framework identifies twelve categories of potential bias in model judges.⁷⁴ Four of them matter most for a regulated evaluation.

The self-preference finding deserves a moment. If the mechanism is familiarity rather than authorship, then choosing a different model as the judge does not eliminate the problem. It shifts which outputs get the benefit. That has a practical consequence: any judge configuration has to be calibrated against human judgment on your own material, and the calibration does not transfer from one deployment to another.⁵

What calibration against human judgment actually requires

The phrase “we calibrated the judge” is doing a lot of unexamined work in current practice. Calibration that means something looks like this:

• A human-scored calibration set drawn from your real inputs, scored under the same rubric the judge will apply, by the same qualified panel, with the human panel’s own inter-rater reliability measured first. You cannot calibrate a judge against a standard that is itself unreliable.
• An agreement measurement between judge and human panel using the same coefficient you used for human-human agreement, so the two numbers are comparable. Judge-human agreement should be reported alongside human-human agreement, never on its own.
• Error analysis by direction and severity, not just an agreement number. A judge that occasionally marks acceptable output as unacceptable creates review work. A judge that marks unacceptable output as acceptable creates escape. Those are different problems with different tolerances, and a single agreement coefficient hides the difference.
• Sensitivity checks on the known failure modes: the same pairs presented in both orders, matched pairs differing only in length, and outputs from more than one generating system.
• A defined re-calibration trigger. The judge model changes, the judge prompt changes, the generating model changes, or the input distribution changes. Any of those invalidates the calibration.
• A stated scope of use. Write down what the judge is authorized to decide and what it is not.

Documenting the Evaluation So It Supports a Qualification File

Everything above is method. This section is about evidence. The difference is that a method produces a result and evidence produces a result somebody else can rely on without repeating the work. In a regulated setting the second is the only one that counts, and it is where most otherwise sound evaluations fall apart.

A useful way to think about the requirement: an assessor should be able to read your file and reconstruct what you did, what you decided in advance, what you found, and why you concluded the system was fit for its intended use. Nothing in that sentence is specific to AI. It is the same expectation that applies to any validation activity, which is convenient, because it means the file structure is already familiar to your quality organization.

The six elements

Sizing the sample without pretending to more precision than you have

There is no published standard sample size for evaluating generative output in a GxP context, and any consultant who gives you one without asking about your use case is selling something. What you can do is make the number defensible.

Start from what the evaluation has to demonstrate. If the primary question is whether the unsupported-statement rate is below a stated threshold, the sample size follows from the precision you need on that proportion and the rate you expect. If the primary question is whether a rare, high-severity failure mode occurs, random sampling is the wrong instrument at any feasible size, and the answer is targeted adversarial testing designed around the failure mode instead. If the question is whether reviewers agree well enough to trust the rest of the evaluation, the overlap set needs to be large enough that the confidence interval around the agreement coefficient is narrow enough to support the criterion.

Stratification usually buys more than raw size. A sample of 150 outputs stratified across document type, input length, source-material quality, and therapeutic area tells you far more than 400 drawn at random from whatever was available, and it lets you report performance by stratum rather than as a single average that conceals a weak segment.

Connecting to the existing framework without overstating it

The temptation is to write in the file that a particular regulation requires this evaluation design. Resist it, because it is not true and it invites a challenge you will lose. What is true, and is enough:

Annex 11 requires validation appropriate to risk, with documented testing evidence and documented assessment of results, and it applies to computerized systems used in GMP-regulated activity in the EU.²⁰ The EMA reflection paper describes current thinking on artificial intelligence across the medicinal product lifecycle and takes a risk-based position on development, deployment, and performance monitoring, while explicitly being a reflection rather than binding guidance.¹⁹ Industry guidance on computerized system validation has extended to cover the AI and machine learning lifecycle, including defining intended use, selecting data, establishing performance measures, and continuous monitoring in operation.

None of that mandates Krippendorff’s alpha at 0.8 or a panel of three. What it does establish is that risk-proportionate, documented, pre-specified evidence of fitness for intended use is the expectation, and that an evaluation designed the way this article describes meets that expectation on its own terms. Write it that way. Say what your evaluation demonstrates and what it does not, cite the framework as the reason you designed it this way rather than as a requirement it satisfies, and you will have a file that holds up in a discussion instead of one that invites an argument about what a guidance document says.

Conclusion

The absence of a reference answer does not make a system unvalidatable. It moves the evidence from correctness against an answer key to agreement among qualified people applying a defined standard, which is a different kind of evidence with a well-developed method behind it. Rubrics that decompose quality into observable attributes, error taxonomies rather than impression scores, inter-rater reliability measured before any results are believed, blinded paired comparison where the real question is relative rather than absolute, and adjudication borrowed from clinical endpoint committees. None of it is exotic. Most of it predates the current generation of models by many years.

What is new is the pressure to substitute automation for the human judgment at the center of it. Reference-free automatic metrics correlate weakly with expert judgment on open-ended text, and model judges, while a genuine advance, carry documented position, verbosity, self-preference, and prompt-sensitivity effects, and show noticeably lower agreement with experts in specialist domains than in general ones. The right use of both is as a screening and monitoring layer with quantified error, feeding a human decision that remains accountable. A validation file that says exactly this, with the calibration data to support it, is stronger than one claiming an automated evaluation the evidence does not yet support.

The work that most often decides whether any of this holds up is the least technical part: an approved protocol before execution, a stated sampling rationale, acceptance criteria fixed in advance, reviewer qualification on file, a results record that shows the failures, and a defined trigger for re-evaluation when the model changes. Sakara Digital works with pharma and biotech organizations building evaluation approaches for generative and summarization systems in GxP settings, including rubric design, panel and adjudication structure, and the protocol and evidence package behind it. If you are working out how to evidence output quality for a system with no answer key, and want an independent perspective on where to start, we are happy to have that conversation.

For Further Reading