Explainability Evidence an Inspector Will Accept for a GxP Model

What the Regulatory Documents Actually Say, and What Status They Hold

Before anyone builds an evidence package, it is worth being precise about which documents exist, what each one actually says about explainability, and what legal weight it carries. Attribution errors are common in this area. Vendors and conference speakers routinely describe “Annex 22 requirements” or “FDA’s explainability rule” in terms the documents do not use. An inspector who has read the text will notice.

Draft EU GMP Annex 22 (still a draft)

The European Commission published draft Annex 22, titled “Artificial Intelligence,” for public consultation on July 7, 2025, alongside a revised Annex 11 and Chapter 4. The consultation closed on October 7, 2025. As of this writing, no final text has been adopted and no implementation date exists. Anything a company builds against Annex 22 today is built against a consultation draft that may change.¹

What the draft says is nonetheless useful, because it is the only GMP text anywhere that names explanation methods directly. Its scope is narrow: computerized systems in manufacturing where AI models are used in “critical applications with direct impact on patient safety, product quality or data integrity, e.g. to predict or classify data.” It covers static models with deterministic output. It states that dynamic models that learn during use, probabilistic models, and generative AI and large language models “should not be used in critical GMP applications.”¹ So the explainability sections of Annex 22 are written for classifiers and predictors, such as a visual inspection model that rejects vials or a model that predicts a critical quality attribute, not for a document-drafting assistant.

Section 8 of the draft is titled “Explainability” and contains two clauses. Clause 8.1, “Feature attribution,” states that during testing of models used in critical GMP applications, “systems should capture and record the features in the test data that have contributed to a particular classification or decision (e.g. rejection),” and that “where applicable, techniques like feature attribution (e.g. SHAP values or LIME) or visual tools like heat maps should be used to highlight key factors contributing to the outcome.” Clause 8.2, “Feature justification,” states that “in order to ensure that a model is making decisions based on relevant and appropriate features and based on risk, a review of these features should be part of the process for approval of test results.”¹

Section 9, “Confidence,” adds two more. Clause 9.1 says the system should, “where applicable, log the confidence score of the model for each prediction or classification outcome.” Clause 9.2 says models “should have an appropriate threshold setting to ensure predictions or classifications are made only when suitable,” and that if the confidence score is very low, “it should be considered whether the model should flag the outcome as ‘undecided’, rather than making potentially unreliable predictions or classifications.”¹

Read carefully, those four clauses describe a specific and fairly modest record: attribution captured during testing, a documented review of whether the attributed features make process sense as part of approving test results, per-output confidence logging, and a defended threshold with an “undecided” path. The draft does not say explanations must be produced for every production decision. It does not name a particular method as mandatory. It does not define what an acceptable explanation looks like. Those gaps are where a company’s own rationale has to do the work.

FDA’s January 2025 draft guidance on AI credibility (still a draft)

FDA’s “Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products” was published in January 2025 under docket FDA-2024-D-4689. FDA’s guidance page still lists it as a Draft Level 1 Guidance, “Not for implementation. Contains non-binding recommendations.”² It applies to AI used to produce information or data supporting regulatory decisions on safety, effectiveness, or quality, which includes manufacturing models.

The word “explainability” does not appear in this guidance at all. In its background section, FDA notes that understanding how AI models are developed and how they arrive at their conclusions “may be difficult and necessitate methodological transparency (e.g., detailing in the regulatory submission the methods and processes used to develop a particular AI model),” and that “uncertainty of the accuracy in the deployed models’ output may be difficult to interpret, explain, or quantify.”³

The operative content is the seven-step risk-based credibility assessment framework: define the question of interest, define the context of use, assess model risk, develop a credibility assessment plan, execute it, document results and deviations, and determine adequacy for the context of use. Model risk is defined as a combination of “model influence” (the contribution of the model’s evidence relative to other evidence) and “decision consequence” (the significance of an adverse outcome from an incorrect decision).³

Within Step 4, the guidance lists what a sponsor should describe about the model: inputs and outputs, architecture, features, the feature selection process and loss functions, parameters, and a rationale for the modeling approach. On evaluation, it asks sponsors to “specify the process by which the uncertainty and confidence level of model predictions were estimated,” notes that “information regarding the uncertainty of model output is important because it helps interpret model outputs,” and states that “all performance estimates should be provided with confidence intervals.” It also says that where the context of use involves a human in the loop, evaluation methods should “consider the performance of the human-AI team, rather than just the performance of the model in isolation.”³ There is no mention of SHAP, LIME, model cards, or any named explanation technique.

The FDA and EMA joint principles (January 2026)

On January 14, 2026, FDA and EMA published “Guiding Principles of Good AI Practice in Drug Development,” ten short principles covering AI used to generate or analyze evidence across nonclinical, clinical, post-marketing, and manufacturing phases. The document describes itself as intended “to lay the foundation for developing good practice” and says the areas of collaboration it describes “may help inform regulatory policies and regulatory guidelines in different jurisdictions.” It is not a guidance and creates no obligations.⁴

Two principles touch explainability directly. Principle 7, “Model design and development practices,” says development should follow best practices in model and system design and use fit-for-use data, “considering interpretability, explainability, and predictive performance,” and that good development “promotes transparency, reliability, generalizability, and robustness.” Principle 10, “Clear, essential information,” says plain language should be used to present “clear, accessible, and contextually relevant information to the intended audience, including users and patients, regarding the AI technology’s context of use, performance, limitations, underlying data, updates, and interpretability or explainability.”⁴ Principle 9 adds that AI technologies “undergo scheduled monitoring and periodic re-evaluation to ensure adequate performance (e.g., to address data drift).”⁴

The ISPE GAMP Guide: Artificial Intelligence (July 2025)

ISPE published the GAMP Guide: Artificial Intelligence in July 2025. It is a 290-page industry good practice guide that sits alongside GAMP 5 Second Edition and covers AI-enabled computerized systems across the life cycle, with a risk-based approach and attention to ongoing monitoring and control.⁵ It is not a regulation. In the authors’ own description of the guide in Pharmaceutical Engineering, “Explainable AI” is presented as a means “to support human-AI-team collaboration in GxP regulated processes,” and “Trustworthy AI” is framed through “transparency, human oversight, and bias mitigation.” The guide’s model governance strategies aim at “decision traceability throughout development and use,” and it emphasizes supplier collaboration and enhanced supplier assessment for AI-enabled systems.⁶ In a later summary, the same authors note that “general principles like the need for transparency, independence of test data sets, and the relevance of ongoing monitoring are becoming an agreed standard,” and say that the factors to consider when retiring a model include “traceability of model input, the model, and model output, as well as the integration into the AI-enabled computerized system to allow for ex-post assessment.”⁷

Context: the EU AI Act and the device-side transparency principles

Two further documents are worth knowing about, with caveats. Article 13 of the EU AI Act requires high-risk AI systems to be “sufficiently transparent to enable deployers to interpret a system’s output and use it appropriately,” and requires instructions for use to include the level of accuracy and its metrics, known limitations, and “where applicable, information to enable deployers to interpret the output.”⁸ Most pharma manufacturing AI is unlikely to be classified as high-risk under the Act, and the high-risk obligations have been deferred (Annex III systems to December 2027 and Annex I systems to August 2028), so this is context rather than a current obligation.

On the device side, FDA, Health Canada, and MHRA published “Transparency for Machine Learning-Enabled Medical Devices: Guiding Principles” in June 2024. It defines transparency as “the degree to which appropriate information about a MLMD (including its intended use, development, performance and, when available, logic) is clearly communicated to relevant audiences,” and lists the “logic of the model, when available” and “device limitations, including biases, confidence intervals and data characterization gaps” among the information that should be shared.⁹ It is a device document and does not apply to pharma manufacturing, but the “when available” qualifier on model logic is the most candid regulatory statement anywhere that logic may simply not be available, and that other evidence must then carry the weight.

Two Readers, Two Explanations: The Data Scientist and the QA Reviewer

Most explainability evidence fails not because the method was wrong but because it was written for the wrong reader. A feature attribution chart is a data scientist’s artifact. It answers a data scientist’s question: which inputs moved this output, and by how much. A QA reviewer or inspector is asking a different question: is this model making its decision for reasons that make sense in this process, and how do I know that is still true today.

What the literature says about the people who run the tools

Kaur and colleagues studied how data scientists actually use two interpretability tools, the InterpretML implementation of generalized additive models and the SHAP package, through a contextual inquiry with 11 participants and a survey of 197. They found that data scientists “over-trust and misuse interpretability tools,” and that few participants could accurately describe the visualizations the tools produced.¹⁰ That is a study of the experts, not the novices. If the people who generate SHAP plots misread them, a QA reviewer handed the same plot without an accompanying narrative has little chance.

Jesus and colleagues ran an application-grounded evaluation with professional fraud analysts, comparing decisions made with raw data only, data plus the model score, and data plus score plus explanations from LIME, SHAP, and TreeInterpreter. The analysts made the most accurate decisions with data only, and every explainer configuration underperformed that baseline, though explanations did help relative to seeing only the score. The authors warn that explainers chosen on proxy metrics like fidelity and stability “might be chosen that, in fact, hurt the overall performance of the combined system of ML model + end-users.”¹¹ For a GxP process with a human in the loop, this is the direct warning: an explanation display can change operator behavior for the worse, and the human-AI team, not the model, is what gets evaluated under the FDA draft framework.³

Underneath both studies is a definitional problem that Lipton described in 2018: “interpretability” is not one property but a bundle of loosely related goals, and a claim that a model is interpretable is nearly meaningless unless someone says interpretable to whom and for what purpose.¹² Doshi-Velez and Kim proposed distinguishing application-grounded evaluation (real people doing the real task), human-grounded evaluation (lay people on simplified tasks), and functionally-grounded evaluation (proxy metrics with no humans).¹³ Almost all explainability evidence produced in pharma today is functionally grounded. A reviewer can reasonably ask for at least one piece of evidence that a process expert used the explanation to reach a correct judgment.

What separates an actionable explanation from a technical one

The difference in every row is the same: the actionable version adds a human judgment, a record of that judgment, and a link to the model as it exists now. Rudin’s 2019 argument goes further: for high-stakes decisions, post hoc explanations of a black box model are themselves a source of risk, because they are approximations of the model rather than the model, and the better path is to build an inherently interpretable model where the problem permits.¹⁴ For many GMP classification problems on tabular data, that is a real option, and a reviewer may reasonably ask why a black box was chosen when an interpretable model would have met the acceptance criteria. The model selection rationale belongs in the evidence package for that reason.

The Evidence Types and Where Each One Breaks

Draft Annex 22 names SHAP, LIME, and heat maps as examples.¹ The FDA draft asks for uncertainty and confidence estimation.³ Neither tells a quality team what the known weaknesses of these methods are, and an inspector who has read the literature may. What follows is each evidence type, what it actually shows, and the published limit that a package should acknowledge rather than hide.

Global feature importance

Global importance ranks input features by their average contribution across a dataset, whether from permutation importance, aggregated SHAP values, or model-native measures such as tree split gain. It answers “what does this model rely on in general.” It is the right first page of an evidence package because a process expert can read it without training: if a vial rejection model’s top feature is a lighting artifact rather than a defect signature, that is visible here. Its limit is that it is an average. A model can rely on sensible features on average and on a nonsensical one for a specific subgroup, which is exactly the case draft Annex 22 clause 3.2 asks companies to examine by dividing the input space into subgroups.¹ Global importance should therefore be reported per subgroup, not only overall.

Local attribution: SHAP and LIME

SHAP, introduced by Lundberg and Lee in 2017, assigns each feature a contribution to a single prediction using Shapley values from cooperative game theory, and unifies several earlier attribution methods under one framework.¹⁵ LIME, introduced by Ribeiro, Singh, and Guestrin in 2016, fits a simple interpretable model in the neighborhood of a single prediction by perturbing the input and observing the output.¹⁶ Both are model-agnostic, both are widely implemented, and both are named in the Annex 22 draft. They are also the methods with the best-documented weaknesses.

Alvarez-Melis and Jaakkola argued in 2018 that a basic requirement of any explanation is that “similar inputs should give rise to similar explanations,” proposed metrics for that property, and showed that current methods “do not perform well according to these metrics.”¹⁷ In practical terms, two nearly identical images of the same vial can yield visibly different attribution maps. Slack and colleagues showed that a classifier can be deliberately constructed so that “post hoc explanations of the scaffolded classifier look innocuous” while the underlying predictions remain biased, and that such classifiers “can easily fool popular explanation techniques such as LIME and SHAP.”¹⁸ The relevance in GxP is not that a vendor is adversarial; it is that a method which can be fooled by construction is also a method that can be misled by ordinary correlation in training data. Kumar and colleagues added that “mathematical problems arise when Shapley values are used for feature importance,” that fixing them requires causal assumptions, and that the resulting explanations may not match what people actually need from an explanation.¹⁹

None of this means SHAP and LIME should be excluded. It means the evidence package should record the method version and settings (background dataset, number of perturbation samples, kernel width), report attribution stability on repeated runs and on slightly perturbed inputs, and state in plain language that these are approximations of the model’s behavior rather than a readout of its internals.

Saliency and heat maps for image models

For visual inspection models, heat maps are the natural explanation and the Annex 22 draft names them.¹ Two papers should temper confidence in them. Adebayo and colleagues proposed “sanity checks” for saliency methods and found that several popular ones produce nearly the same map whether the model is trained or has randomized weights, meaning the map reflects the image’s edges rather than what the model learned.²⁰ Ghorbani, Abid, and Zou showed that “interpretation of neural networks is fragile”: imperceptible changes to an input can produce a very different saliency map while leaving the prediction unchanged.²¹ A package that includes heat maps should state which saliency method was used, whether it passed the randomization sanity check, and whether maps for near-duplicate images agree. If nobody has run those checks, that is a finding waiting to happen.

Decision boundary and threshold documentation

For a classifier, the single most consequential design decision is where the threshold sits, and it is the piece of explainability evidence quality teams most often leave out because it does not look like an “explanation.” Annex 22 draft clause 9.2 asks for “an appropriate threshold setting” and for consideration of an “undecided” outcome at low confidence.¹ The record should show the threshold value, the operating characteristic it was chosen on (sensitivity versus specificity trade-off, with the confusion matrix at that point), who approved it, the rationale linking it to the process (for example, that false accepts are the failure that matters and false rejects go to a manual re-inspection), the “undecided” band and what procedure handles it, and per-subgroup performance at the chosen threshold. A reviewer can act on that. A reviewer cannot act on “the model uses a 0.5 threshold.”

Worked example libraries

A worked example library is a curated set of representative inputs and the model’s outputs, each with attribution, confidence, and a process expert’s annotation of whether the decision and its reasons are correct. It should span the subgroups defined for intended use, include the rare variations Annex 22 draft clause 3.1 asks companies to characterize, and include cases at and near the threshold.¹ This is the evidence type closest to how an inspector already thinks: show me a case, walk me through it. It is also the evidence type that best supports the Annex 22 draft’s “feature justification” review, because the justification is done case by case by a named expert and signed. Its limit is coverage. A library of forty cases proves the model behaved sensibly on forty cases; it should be presented as a qualitative complement to the quantitative test, never a substitute.

Counterfactual examples

Wachter, Mittelstadt, and Russell proposed counterfactual explanations in 2017 as a way to explain an automated decision without opening the model: state the smallest change to the input that would have changed the output.²² For a GMP model this translates to statements such as “this unit would have been accepted if the measured fill volume had been 0.3 mL higher” or “this batch would not have been flagged if the temperature excursion had been 4 minutes shorter.” Counterfactuals are unusually useful for QA review because they are testable: a process expert can say whether the stated change is physically meaningful. Their limit is that many counterfactuals exist for any decision, some of them physically impossible, so the package should state how the counterfactuals were generated and constrained to realistic input ranges.

Uncertainty and confidence reporting

The Annex 22 draft asks for confidence scores to be logged per outcome.¹ The FDA draft asks sponsors to specify how uncertainty and confidence were estimated and to report performance with confidence intervals.³ The trap is that a softmax output is not a calibrated probability. Guo and colleagues showed in 2017 that “modern neural networks, unlike those from a decade ago, are poorly calibrated,” and that a simple post-processing step, temperature scaling, “is surprisingly effective at calibrating predictions.”²³ A confidence threshold set on an uncalibrated score is a threshold on a number that does not mean what the procedure says it means. The evidence should include a calibration plot (reliability diagram) and the calibration method, or an explicit statement that the score is a ranking rather than a probability and that the threshold was set empirically on test data.

For teams that want a confidence statement with a guarantee, conformal prediction offers one. Angelopoulos and Bates describe it as a distribution-free method that wraps any model and produces prediction sets with a user-chosen coverage level, under an exchangeability assumption.²⁴ In a classification setting, a conformal prediction set that contains both “accept” and “reject” is a principled version of the Annex 22 draft’s “undecided” flag, with a stated error rate rather than an arbitrary cutoff. That assumption of exchangeability is also why it must be re-checked after any change in input distribution.

Model cards and equivalent summaries

Mitchell and colleagues proposed model cards in 2019 as short documents accompanying trained models that report intended use, out-of-scope uses, evaluation data, performance disaggregated by relevant conditions and groups, and ethical considerations.²⁵ In a GxP setting, the model card is the cover sheet of the evidence package: one or two pages a reviewer can read in five minutes that state what the model is for, what it is not for, how it performed on which data, and where the detailed evidence lives. It should not carry any claim not supported by a document behind it. None of the regulatory documents reviewed above requires a model card by that name, and the package should not claim otherwise; it should present the card as the company’s chosen way of meeting the joint principles’ call for “clear, accessible, and contextually relevant information.”⁴

Tying the Evidence to Intended Use and Risk Classification

Every document reviewed above makes the same structural move: define intended use first, then assess risk, then scale the evidence to the risk. Explainability evidence is no exception. The depth of the package should be an output of the risk assessment, not a fixed template applied to every model.

Intended use defines what an explanation is for

Draft Annex 22 clause 3.1 asks for the intended use to be described “in detail,” including “a comprehensive characterisation of the data the model is intended to use as input and all common and rare variations,” with a process subject matter expert responsible for the adequacy of that description. Clause 3.3 adds that where a model gives input to a human decision and the testing effort has been reduced on that basis, the description “should include the responsibility of the operator.”¹ The FDA draft’s Step 2 asks for the context of use, and Step 1 for the question of interest the model helps answer.³

That description determines the explanation audience. If the model is fully automated with a downstream release test as the safeguard (the FDA draft’s own fill-volume example, where release testing reduces model influence and lowers model risk to medium despite high decision consequence³), the explanation audience is the QA reviewer doing periodic review and the investigator handling a deviation. If the model presents a recommendation to an operator who makes the call, the explanation audience is the operator in real time, and the evidence must include what the operator sees, how they were trained to read it, and how the human-AI team performed in testing, which is what the FDA draft explicitly asks for.³ The Jesus study is the reason this matters: an explanation display can lower the accuracy of the combined system.¹¹

Risk sets the depth

Using the FDA draft’s two axes, model influence and decision consequence, a practical scaling looks like this:

The risk classification itself has to be in the package, with the reasoning. An inspector who sees a thin explainability record for a model that rejects product will want to know whether the thinness was a decision or an oversight. A one-page risk rationale that cites model influence, decision consequence, and the mitigating controls turns a potential observation into a documented judgment call.

When the Model Belongs to a Vendor and the Internals Are Closed

Much of the AI in GMP manufacturing arrives inside a purchased system: a vision inspection platform, a predictive maintenance module, a chromatography data system feature. The regulated company often cannot see the architecture, the training data, or the weights. Draft Annex 22 clause 2.2 does not accept that as a reason for a thinner record. It states that documentation “should be available and reviewed by the regulated user irrespective of whether a model is trained, validated and tested in-house or whether it is provided by a supplier or service provider.”¹ The GAMP AI Guide likewise emphasizes enhanced supplier assessment and collaboration for AI-enabled systems.⁶

What to ask the supplier for

The request list should mirror the FDA draft’s Step 4 description items, because they are the clearest published statement of what a reviewer expects to see about a model: inputs and outputs, architecture class, features, feature selection, parameters (or at least a statement of their count and governance), training data description and how labels were established, the evaluation approach, performance with confidence intervals, and how uncertainty was estimated.³ Suppliers can provide most of that without disclosing weights or proprietary training sets. Where they refuse, the refusal should be documented, and the gap closed with evidence the regulated company generates itself.

Evidence the regulated company can generate without the internals

Every explanation type in the table above except native model-specific importance can be produced against a closed model, because SHAP’s kernel variant, LIME, counterfactual search, and calibration all work on inputs and outputs alone. That is their entire reason for existing. So a closed vendor model does not prevent an explainability record; it changes who produces it and shifts the emphasis toward behavioral evidence:

• Independent test data and independent metrics. The Annex 22 draft’s sections 5 through 7 on test data, independence, and execution apply in full, and the regulated company owns them regardless of where the model came from.¹
• Behavioral attribution. Model-agnostic attribution on the company’s own test set, with the stability checks described earlier, so that the “feature justification” review can still be performed by a process expert.
• A worked example library built from the company’s own product and lines, since the supplier’s examples were not drawn from the intended use as this site defines it.
• Counterfactual probing at the decision boundary, which is often the fastest way to discover that a closed model is sensitive to something it should not be.
• Confidence logging and calibration on site data, with the threshold set and approved by the regulated company, not accepted as a vendor default.
• Contractual commitments on change notification, so that a silent model update in a software release does not invalidate the record. The Annex 22 draft’s change control and configuration control clauses (10.1 and 10.2) require the company to detect unauthorized change; it cannot do that if the supplier does not tell it what changed.¹

The device-side transparency principles are worth borrowing here for their honesty: model logic is to be communicated “when available.”⁹ A package for a closed vendor model should say plainly that the logic is not available, explain what was requested and what was received, and show that the behavioral evidence covers the intended use. That is a defensible position. Pretending the vendor’s marketing white paper is an explainability record is not.

Keeping the Evidence Current After Retraining

An explainability record is a statement about a specific model version on a specific test set with a specific explanation tool. Any of the three can change. The regulatory documents agree on the response. Draft Annex 22 clause 10.1 requires a tested model, its system, and the whole process to be under change control before deployment, and says any change to the model, the system, the process, or the physical objects it takes as input “should be documented and evaluated to determine if the model needs to be retested,” with any decision not to retest “fully justified.”¹ The FDA draft says that depending on the extent and impact of a change, “some steps in the credibility assessment plan may need to be re-executed, including retraining and retesting the model,” and asks for life cycle plans that specify performance metrics, a risk-based monitoring frequency, and “triggers for model retesting.”³ The joint principles call for “scheduled monitoring and periodic re-evaluation.”⁴

Retraining as a change control event with an explanation clause

Most organizations already treat retraining as a change. Fewer treat the explanation record as part of what the change affects. A practical procedure adds four things to the retraining change record:

1. Regenerate every explanation artifact against the new model version using the same test set version and the same explanation tool version, and file them as a new revision, never overwriting the old.
2. Compare the new global importance and subgroup importance with the last approved set, and require a process expert to review and sign off on any material change in the ranking. A model that now depends on a different feature to reach the same accuracy has changed in a way the accuracy metric cannot see.
3. Re-run the worked example library and flag every case where the decision, the attribution, or the confidence changed. Cases that flipped are the first things a reviewer will ask about.
4. Re-check calibration and, if conformal methods are used, re-check coverage, since both depend on the relationship between model and data that retraining alters.^23,24

Explanation drift as a monitoring signal

Annex 22 draft clauses 10.3 and 10.4 require monitoring of model performance metrics and of whether input data remain within the sample space, with defined drift metrics.¹ Attribution offers a third monitoring channel that neither of those captures: the distribution of attributed features on production inputs. If a vision model that was approved because it attends to defect regions begins attending to background regions on a new lot of containers, performance metrics may not move for weeks, and the input distribution may look within range on the monitored variables. A periodic sample of production attributions compared against the approved reference set will catch it. This is not required by any of the documents reviewed here; it is a control that follows naturally from having built the record in the first place, and it converts an explainability package from a validation artifact into an operational one.

The Evidence Package: A Proposed Table of Contents and the Inspector’s Questions

The following is a proposed structure for an explainability evidence package for a validated AI or machine learning model used in a GxP decision. It is intended to sit within, or be cross-referenced from, the system’s validation package under Annex 11 or the site’s computer software assurance approach, and to be scaled by risk as described above. Section numbers are suggestions.

Proposed table of contents

Questions an inspector is likely to ask

These are framed the way they tend to be asked: plainly, in sequence, each one following from the last. The package section that should answer each is noted.

The last question is not a trick. It is a check on whether the people presenting the package understand what they are presenting. The correct answer, given plainly, builds more credibility than any chart in the binder.

Conclusion

The regulatory texts on explainability are converging on a shared picture even while most of them remain unfinished: know the intended use, capture what the model relies on, have a process expert judge whether that is sensible, log and threshold confidence honestly, evaluate the human and model together where there is a human, and keep all of it current through change control and monitoring. What the texts do not supply is a warning about the tools. The peer-reviewed literature supplies that in abundance: attribution methods are unstable, can be misled, are misread by the experts who run them, and can make human decisions worse when displayed carelessly. An evidence package that takes both sources seriously looks different from one built to satisfy a checklist. It is organized around questions a reviewer would ask, it records judgments and who made them, it states the limits of its own methods, and it can be regenerated and compared after the model changes.

Sakara Digital works with pharma and biotech organizations building validation and governance records for AI and machine learning models in GxP processes, including the explainability evidence that has to survive a careful reviewer. If you are preparing a model for inspection, assessing a vendor system whose internals you cannot see, or deciding how much explanation evidence a given model actually needs, we are happy to have that conversation.

For Further Reading