Why the Demographics Checklist Is Not Enough

When bias testing entered the mainstream of pharmaceutical AI development, it arrived in a form that was borrowed almost intact from consumer-facing algorithmic auditing. Split the population by age, sex, and race, calculate the metric of choice in each cell, and confirm that the values fall within a stated tolerance. The approach is defensible when the model is a credit-scoring system and the sensitive attributes are the ones the law names. It is much less defensible when the model recommends who to enroll in an oncology trial, which patients should be flagged for a rare disease workup, or how to weight prescriber signals in a safety surveillance pipeline.

The core problem is that a demographics-only view mistakes visibility for causation. Race is not what makes a model behave differently across racial groups. The underlying drivers are the data-generating processes that produced the training set: which patients showed up in the historical trials, which physicians prescribed which drugs to whom, which hospitals kept the cleanest structured records, and which observations were coded consistently over time. A bias assessment that ignores these mechanisms may satisfy a checklist but will not satisfy a reviewer who understands the domain.

Regulators have caught up quickly. A 2026 critical review of the FDA’s draft guidance on artificial intelligence in drug and biological product regulation flagged the absence of specific metrics for bias assessment and the limited guidance on addressing historical bias in training data as one of the most significant remaining gaps.2 That gap is closing. The Diversity Action Plan requirements introduced under the Food and Drug Omnibus Reform Act of 2022 (FDORA) now shape how sponsors think about representation across the whole product lifecycle, not just at enrollment.3 Bias testing sits directly at that intersection.

28.7% of clinical evaluations provide sex-specific data in AI-enabled device studies2
23.2% address age-related patient subgroups in comparable analyses2
95% of real-world evidence studies contain at least one avoidable methodological issue known to incur bias4

The message from these numbers is not that the industry is careless. It is that the standard checklist misses categories of skew that are structural, and structural bias is harder to detect than distributional bias. That is the terrain the next generation of testing has to cover.

Six Types of Bias Specific to Pharma AI

The following six categories cover the bias patterns we see most consistently in pharmaceutical AI work. They overlap, and any given model will usually carry more than one. Naming them separately matters because the statistical treatment differs from one to the next.

1. Historical Clinical Trial Exclusion Bias

Every model that touches drug efficacy, dose response, adverse event risk, or trial eligibility inherits the exclusion patterns of the trials that generated the training data. Pivotal trials have historically underrepresented older patients, women in certain therapeutic areas, and racial and ethnic minorities across nearly every indication. A model trained on those endpoints will produce more confident and better-calibrated predictions for the populations that were actually studied and less reliable predictions for those who were not.5

The consequence is subtle. The model may not obviously misclassify underrepresented patients. It may simply be more uncertain about them, and if the downstream decision logic treats uncertainty as an implicit exclusion criterion, the model quietly reproduces the original enrollment bias. Detecting this pattern requires comparing not just point estimates but calibration and confidence intervals across subgroups.

2. Indication Drift

A model built on one indication frequently gets pushed sideways into an adjacent one. The oncology response predictor trained on non-small cell lung cancer gets applied to squamous variants. The safety signal model tuned on chronic use gets used in acute settings. Every drift of this kind is a bias risk, because the population characteristics that made the model perform well on the original indication may not hold on the new one. Indication drift is one of the most common sources of failure we see in production pharma AI, and it rarely triggers a demographic subgroup alarm because the demographics on the new indication may look similar to the old one.6

3. Site-Level Bias

In multi-site studies and real-world evidence analyses, the site itself carries information. Sites differ in their patient mix, referral patterns, standard-of-care protocols, and documentation habits. A model that learns to lean on site as a proxy variable will generalize poorly to any new site, and the mismatch will not be captured by patient-level demographics. The APPRAISE framework for evaluating real-world evidence explicitly names site-level selection as a bias domain that requires targeted assessment.7

4. Prescriber Pattern Bias

Prescriber behavior is a systematic signal in almost any pharmaceutical data set built from insurance claims or EMR extracts. Physicians vary in what they prescribe, when they escalate therapy, and how they document response. When those patterns correlate with patient characteristics, and they usually do, the model will encode prescriber preferences as if they were features of the patient. This is the class of bias behind under-prescription of effective medications to older patients that has been demonstrated in causal fairness studies.8

5. Temporal Bias From Real-World Data

Real-world data is not a snapshot. It is a moving stream, and the stream changes in structured ways. Coding practices shift with new billing codes. Documentation improves as EMR systems mature. Standard of care evolves. Any model trained on a fixed window and deployed forward is subject to temporal drift, and the drift is often correlated with the very attributes the fairness assessment is trying to hold constant. Data drift monitoring is now considered a baseline requirement in FDA guidance for AI-enabled medical devices, but the same principle applies to non-device pharmaceutical AI.9

6. Missingness Bias in EMR Data

EMR data does not miss uniformly. Missingness rates for some variables exceed 80%, and up to 40% of the predictive signal in some clinical models comes from the missingness pattern itself rather than from the recorded values.10 When missingness correlates with race, socioeconomic status, or site quality, the model learns those correlations and reproduces them under the label of clinical judgment. This is the most insidious form of pharma AI bias because the underlying data looks clean.

Why the taxonomy matters. Each of these categories responds to a different statistical treatment. Historical exclusion is a calibration and coverage problem. Indication drift is a distribution shift problem. Site and prescriber bias are hierarchical modeling problems. Temporal drift requires monitoring rather than a one-time audit. Missingness bias requires principled handling in the feature pipeline. Lumping them all into a demographic subgroup table erases the very structure that would let you fix them.

The Statistical Toolkit for Bias Testing

The statistical methods that support serious bias testing are not new, but they are unevenly adopted in pharmaceutical AI programs. The point of this section is to name the tools that actually work, describe what they detect, and note where they fall short.

Disparate Impact Ratios

The four-fifths rule from employment law has been widely adopted in algorithmic fairness. A disparate impact ratio below 0.8 or above 1.25 is treated as a flag for potential bias.11 In pharma the ratio is useful when the outcome is binary and the exposure is a downstream action. Its limits are well known. It captures unequal outcome rates but not unequal error rates, and it does not distinguish between structural underrepresentation and legitimate clinical difference. Use it as a screen, not as a verdict.

Calibration Slope and Intercept

Calibration is arguably the most important fairness property in a clinical prediction model, and it is systematically underreported. A model can be well-calibrated in aggregate and miscalibrated in every meaningful subgroup, or the reverse. The recommended practice is to report calibration slope and intercept separately for each subgroup of interest, and to look for departures from a slope of 1.0 and an intercept of 0.12 Miscalibration by subgroup is a stronger signal than a raw disparate impact ratio because it maps directly to clinical decision-making.

Subgroup ROC and Precision-Recall Analysis

Standard performance metrics disaggregated by subgroup are the workhorses of bias testing. The AUC-ROC, precision, recall, and F1 score computed within demographic and clinical strata reveal where the model performs and where it does not. The important step, and the one most often skipped, is confidence interval reporting. A model that appears to perform equally across groups at the point estimate may show wide, overlapping confidence intervals in the smaller subgroups. That uncertainty is itself a bias finding.13

Causal Fairness Metrics

Statistical fairness metrics answer the question of whether outcomes differ across groups. Causal fairness asks whether they would differ if a specific attribute were counterfactually changed while everything else were held constant. Counterfactual fairness is technically demanding but increasingly considered the gold standard in domains where the sensitive attribute is confounded with clinical factors. In pharmaceutical AI, causal fairness methods have identified prescription patterns that reflect provider preference rather than clinical evidence, and correcting for those causal biases has been shown to improve treatment outcomes by meaningful margins.8

Group Versus Individual Fairness

Group fairness metrics assess whether populations are treated equally in aggregate. Individual fairness asks whether similar individuals receive similar predictions. Both matter. Pharma teams tend to overweight group metrics because they are easier to communicate. Individual fairness assessments are more computationally demanding but they catch a category of local bias that group metrics smooth over.14

MethodWhat It DetectsBest Use CaseCommon Limit
Disparate impact ratioUnequal positive outcome rates across groupsBinary downstream action with clear positive/negativeSilent on error type distribution
Calibration slope and interceptSystematic miscalibration within subgroupsRisk prediction and probability outputsRequires sufficient sample per group
Subgroup ROC / PR curvesDiscrimination performance gapsRanking and classification modelsConfidence intervals often ignored
Causal fairness / counterfactualEffect of sensitive attribute holding all else equalWhen bias is confounded with clinical variablesRequires assumed causal graph
Individual fairness metricsLocal inconsistency across similar patientsPersonalized medicine and rare diseaseComputationally expensive
Missingness pattern analysisWhether missingness carries predictive signalEMR-derived featuresInterpretation is context-dependent

Designing a Bias Testing Protocol

A bias testing protocol is not just an analysis plan. It is a document that tells a reviewer why you chose the methods you chose, how you selected the reference populations, what thresholds you set in advance, and how you will act on the findings. The protocol should be written before the analysis is executed, for the same reason that a statistical analysis plan is written before the primary efficacy read-out.

1

Context of Use Statement

Define exactly what the model does, for whom, at what decision point, and with what human-in-the-loop safeguards. The context of use anchors every subsequent choice in the protocol. Bias only has meaning against a stated purpose.

2

Reference Population Definition

Specify the population against which fairness is being assessed. This is rarely the training population. It is usually the intended deployment population, and the two can differ in important ways. Document the mapping.

3

Subgroup Prespecification

List the subgroups that will be analyzed and the rationale for each. Include the standard demographic axes, but also add clinically meaningful strata: comorbidity clusters, sites, prescriber archetypes, time windows, and missingness profiles.

4

Metric Selection With Thresholds

Choose the metrics that map to the decisions the model influences. Set numeric thresholds for what counts as a finding requiring action, what triggers documentation only, and what is considered acceptable variation. Prespecification protects against retrofitting acceptance criteria to the results.

5

Monitoring Plan

Bias testing is not one-shot. Include the plan for continued monitoring after deployment, including which metrics will be recomputed, at what frequency, and what will trigger a formal reassessment. This is what regulators mean by algorithmovigilance.15

6

Remediation Or Disclosure Pathway

Decide in advance what the sponsor will do with different classes of findings. Some findings warrant retraining. Some warrant restricted labeling. Some warrant transparent disclosure with continued deployment. The choices should be documented and defensible.

Documentation for FDA and EMA

Bias testing documentation should read like a well-written statistical section of a submission. FDA reviewers are looking for a specific pattern: a clear description of the model, an explicit statement of intended population, a prespecified analysis plan, executed results with numeric findings, and a discussion that addresses both strengths and limitations. Everything else is secondary.

The FDA’s 2025 draft guidance on AI in drug and biological product regulation and the joint FDA-EMA Guiding Principles of Good AI Practice both emphasize traceable documentation, pre-specified context of use, data provenance, and bias control as core evidentiary requirements.16 The reflection paper issued by the EMA specifically calls out training data provenance and bias mitigation as areas where sponsors are expected to demonstrate active management, not just an acknowledgment of risk.17

What a Well-Structured Bias Documentation Package Includes

SECTION 1

Data Provenance and Representativeness

Describe the origin of every dataset used for training, tuning, and validation. Quantify representativeness against the intended deployment population. Name the gaps rather than smoothing them over.

SECTION 2

Bias Testing Protocol

Include the prespecified plan as a numbered appendix. Reviewers should be able to read the protocol before they read the results, and the results should map one-to-one against the plan.

SECTION 3

Quantitative Results by Subgroup

Report all prespecified metrics with confidence intervals. Include the metric values for each subgroup, the reference comparison, and the threshold used to flag a finding.

SECTION 4

Interpretation and Actions

For every finding, state the interpretation, the causal hypothesis, and the action taken. Where no action was taken, explain why the sponsor considered the finding acceptable and how it will be monitored.

SECTION 5

Predetermined Change Control

For SaMD and AI-enabled devices, integrate bias monitoring into the Predetermined Change Control Plan. Specify the retraining triggers, revalidation thresholds, and disclosure requirements.18

SECTION 6

Labeling Implications

Where bias testing reveals limits of the model’s generalizability, connect those limits explicitly to labeling. Underrepresented populations should be reflected in indication language or contraindications where appropriate.

SD perspective. The most common documentation failure we see is not lack of analysis. It is disconnect. The bias testing sits in one section, the labeling sits in another, and the change control plan lives in a third. Reviewers notice the seams. A well-structured package treats bias testing as a thread that runs through provenance, protocol, results, labeling, and post-market monitoring. When those pieces align, the submission tells one coherent story.

Common Gotchas That Trip Up Bias Assessments

Every bias assessment we have reviewed has at least one of the following patterns. None of them are intentional, and most of them are easy to correct once they are named.

Gotcha 1: Using training data as the fairness reference. Fairness is assessed against the intended deployment population, not the training set. If your training data underrepresents the deployment population in exactly the way you are worried about, calculating fairness on the training set will hide the problem.

Gotcha 2: Ignoring confidence intervals. Small subgroups produce wide confidence intervals. A subgroup AUC of 0.72 with a confidence interval from 0.55 to 0.89 tells you almost nothing. Reporting the point estimate without the interval creates a false impression of parity.

Gotcha 3: Confounding sensitive attributes with clinical variables. Race and sex correlate with disease prevalence, standard of care, and outcome distribution. A model that predicts different outcomes for different groups may be reflecting clinical reality or reflecting historical inequity. Statistical fairness metrics cannot distinguish between the two; causal fairness methods can.19

Gotcha 4: Dropping missing data. Complete case analysis is a form of selection that reintroduces the exact bias the assessment is trying to detect. Missingness has to be modeled, not deleted.

Gotcha 5: Treating fairness as static. A model that was fair at validation can drift out of fairness within months of deployment. Without ongoing monitoring, the original assessment ages into an artifact rather than evidence.

Gotcha 6: Optimizing bias metrics against each other. Some fairness definitions are mathematically incompatible. Equal opportunity, predictive parity, and calibration cannot all hold simultaneously in a non-trivial model. Choose the definitions that matter for your context of use and document the tradeoff.

Gotcha 7: Assuming that adding demographic features reduces bias. Including race as a model feature does not remove bias, and may amplify it. The right question is not whether the feature is present but whether the model uses it in a way that is clinically justified.

A Bias Testing Checklist for Pharma AI

The checklist below is a working artifact, not a template. Adapt the items to the model’s context of use, the intended population, and the regulatory pathway. A model in a Diversity Action Plan-eligible pivotal trial will have different requirements than a claims-based safety signal detector.

Data and Design

  • Training, tuning, and validation populations are described with counts and demographics.
  • The intended deployment population is defined and mapped to the training population.
  • Historical clinical trial exclusion patterns in the training data are documented.
  • Missingness patterns are characterized by variable and by subgroup.
  • Site-level and prescriber-level clustering is described where applicable.
  • Temporal coverage is stated with the earliest and latest observation dates.

Testing Protocol

  • The bias testing protocol is prespecified and dated before analysis execution.
  • Subgroups are prespecified, including standard demographics and clinically meaningful strata.
  • Metrics are selected to match the model’s decision function.
  • Numeric thresholds for findings are prespecified.
  • Confidence interval methods are stated and appropriate for subgroup sample sizes.
  • Causal fairness methods are considered where confounding is likely.

Execution

  • Every prespecified analysis is executed and reported, including null findings.
  • Subgroup calibration is reported with slope and intercept.
  • Subgroup discrimination is reported with confidence intervals.
  • Missingness is modeled rather than dropped.
  • Disparate impact ratios are calculated where applicable.
  • Individual fairness is assessed where model outputs drive personalized decisions.

Documentation and Post-Deployment

  • Results are traceable from raw output to summary tables.
  • Every finding is interpreted with an explicit causal hypothesis.
  • Actions taken and not taken are both documented.
  • Labeling implications are described and reflected in draft label language.
  • A predetermined change control plan integrates bias monitoring.
  • Monitoring metrics, frequencies, and reassessment triggers are defined.

Remediation Versus Disclosure: A Decision Framework

Not every bias finding calls for remediation, and not every disclosure is honest without further mitigation. The following framework helps sponsors decide which pathway fits which finding. It is not a substitute for judgment, but it structures the conversation.

PATHWAY A

Remediate Before Deployment

The finding is material to clinical decision-making, is remediable by additional data collection, retraining, or model architecture changes, and the deployment timeline can absorb the delay. Standard when the affected population is intended to use the model.

PATHWAY B

Deploy With Restricted Labeling

The finding is material but not fully remediable with available data. The model is deployed with an indication that excludes the affected population, and the label documents the limitation. Regulators generally accept this when the disclosure is prominent and the limitation is causally explained.

PATHWAY C

Deploy With Ongoing Monitoring

The finding is potentially material but the evidence is limited by sample size or study design. The model is deployed with heightened monitoring in the affected population, and monitoring results feed a formal reassessment at a prespecified point.

PATHWAY D

Transparent Disclosure Only

The finding reflects a real but clinically justified performance difference across groups. The model is deployed with clear documentation of the finding, an explicit causal explanation, and no restriction on use. This pathway is defensible only when the causal explanation is grounded in clinical evidence, not speculation.

Choosing the pathway. The four pathways form a hierarchy of preference for most sponsors and most regulators. Remediation is the default when it is feasible. Restricted labeling is the standard fallback. Ongoing monitoring is appropriate when the finding is real but the evidence is thin. Transparent disclosure alone is the least common pathway and should be reserved for cases where the sponsor can defend the underlying clinical reasoning.

The framework above assumes that the sponsor has genuine choices. In many cases the choice is constrained by data availability, regulatory pathway, or timeline. When it is, the constraint itself should be documented. Reviewers respond well to sponsors who can articulate why they made the choice they made, and less well to sponsors who present the outcome as if it were the only option.

Conclusion

Bias testing in pharmaceutical AI is transitioning from a compliance checkbox into a discipline with its own methodology, protocol structure, and regulatory expectations. The teams that treat it that way are the ones producing submissions that hold up under review, models that generalize to the populations they were built for, and evidence packages that survive the transition from clinical development into commercialization. The teams that continue to run a demographic subgroup table and call it a bias assessment are increasingly finding that reviewers, journal editors, and internal governance boards ask for more.

The pattern we see most consistently in successful programs is not technical sophistication, though that helps. It is intellectual honesty about what the data can and cannot support. A model that has been rigorously tested and found to have documented limits in a specific subpopulation is a stronger submission than a model that has been superficially tested and found to be perfectly fair everywhere. Regulators know which one is which.

Sakara Digital works with pharma and biotech organizations building this kind of AI governance and bias assessment infrastructure. If you are designing a bias testing protocol for a submission, structuring a change control plan for an AI-enabled device, or working out how to document the tradeoffs in a model that has known limits, we are happy to have that conversation.