Building Scientist Trust in AI Output: Review Practices That Work in the Lab

Why Accuracy Numbers Do Not Earn Scientist Trust

Most AI rollouts in R&D start with a slide that shows an accuracy number. The literature summarizer scored well on a benchmark. The assay interpretation model agreed with expert calls most of the time. The ELN search assistant found the right experiment in most test queries. Then the tool goes live, and usage drops off within weeks. When someone asks the scientists why, the answer is rarely that the tool was bad. It is that they could not tell when it was wrong.

That distinction matters. A scientist’s job is to make claims they can defend. A medicinal chemist who picks a scaffold, a bioanalytical scientist who accepts a calibration curve, or a translational lead who summarizes the evidence for a target will be asked to explain the basis for that decision. An aggregate accuracy number does not help them explain anything. If the tool is right 92% of the time, the scientist still has to know which 8% they are holding.

Researchers Are Using AI More and Expecting Less of It

Survey data shows this tension clearly. Wiley’s ExplanAItions study, based on 2,430 researchers surveyed in August 2025, reported that overall AI tool use rose from 57% in 2024 to 84% in 2025. In the same period, concern about inaccuracies and hallucinations rose from 51% to 64%. The year before, researchers believed AI already outperformed humans on more than half of the use cases presented to them; in 2025 that share dropped to less than one third.¹

The same release noted that 57% of researchers cited a lack of guidelines and training as a main barrier to wider adoption, and only 41% felt their organization gave them adequate support.¹ Put those numbers together and the picture is clear. Scientists are using these tools, often on their own, and they have become more skeptical as they have used them more. The gap is not enthusiasm. The gap is a way of working that lets them check the output and decide how far to rely on it.

The Goal Is Appropriate Reliance, Not Maximum Trust

It is tempting to frame the problem as “getting scientists to trust AI.” That framing is wrong, and scientists can tell. The human factors research on automation has long described the target differently. John Lee and Katrina See’s widely cited review is titled, plainly, “Trust in Automation: Designing for Appropriate Reliance.” Its abstract opens with the observation that automation often causes problems because people fail to rely on it appropriately, and that trust guides reliance when complexity and unexpected situations make a complete understanding of the automation impractical.²

That last point describes the lab well. No bench scientist will fully understand how a large language model or a deep learning property predictor reaches its outputs. They will rely on it, or not, based on what they can see and what they have experienced. So the leadership question is not how to raise trust. It is how to make trust match the tool’s real performance on each kind of task, so that scientists rely on it where it has earned that and check it hard where it has not.

What the Research Says About Trust, Reliance, and Automation Bias

Before designing lab practices, it helps to know what decades of research on people working with automated aids have found. The findings are consistent, and some of them are uncomfortable for anyone hoping that training or explanations will solve the problem on their own.

Automation Bias Affects Experts, and Training Alone Does Not Prevent It

Raja Parasuraman and Dietrich Manzey reviewed the empirical work on complacency and automation bias and proposed an integrated model. Their review found that automation bias leads to both omission errors (missing a problem because the aid did not flag it) and commission errors (acting on a wrong recommendation) when decision aids are imperfect. It occurs in both naive and expert participants, affects teams as well as individuals, and, in their words from the abstract, cannot be prevented by training or instructions.³

That finding should change how R&D leaders think about rollout. A training session that tells scientists to “always check the AI” is necessary but nowhere near enough. The working environment has to make checking easy, expected, and visible.

Bias Is Worst When Checking Is Hard

David Lyell and Enrico Coiera’s systematic review screened 890 papers and included 40. Contrary to the earlier human factors view that automation bias mainly shows up under multitasking, they found it in single tasks, typically diagnostic tasks rather than monitoring ones, and where verification was complex. They concluded that automation bias appears to be associated with the cognitive load of the decision task, and that strategies to reduce it might focus on lowering that load.⁴

Diagnostic tasks with complex verification describe a great deal of R&D work: interpreting a dose-response curve, judging whether an image shows a real phenotype, deciding whether a literature claim supports a target hypothesis. The practical lesson is direct. If it takes a scientist twenty minutes to verify an AI answer, they will verify fewer answers. Every practice in this article is partly about making verification faster.

Wrong Suggestions Mislead Experienced Readers Too

A 2023 study in Radiology by Dratsch and colleagues gives one of the clearest pictures of the effect among trained professionals. Twenty-seven radiologists read 50 mammograms with category suggestions presented as coming from an AI system. In 12 of the 40 cases in the second set, the suggested category was incorrect. According to the RSNA’s summary of the study, inexperienced readers assigned the correct category in almost 80% of cases when the AI suggestion was correct, and in less than 20% when it was incorrect. Very experienced readers, with more than 15 years of experience on average, dropped from 82% to 45.5%.⁵

The authors recommended safeguards, including showing users the confidence levels of the decision support system, teaching users how the system reaches its outputs, and making users feel accountable for their own decisions.⁵ Each of those maps onto a lab practice covered below.

Performance Is Uneven Across Tasks That Look Similar

Fabrizio Dell’Acqua and colleagues ran a preregistered experiment with 758 consultants at Boston Consulting Group. On 18 realistic tasks within the capability of the AI, consultants using GPT-4 completed 12.2% more tasks, completed them 25.1% more quickly on average, and produced significantly better work. On one complex task selected to fall outside that capability, consultants using AI were 19% less likely to produce correct solutions than those without it.⁶

The authors call this a jagged frontier: AI help improves some tasks and worsens others, even inside the same workflow and at a similar apparent level of difficulty. For an R&D team, this is the strongest argument against a single accuracy number. A literature assistant may be excellent at summarizing a well-studied pathway and poor at reporting negative results, or at distinguishing a mouse finding from a human one. Scientists need to know where those edges are, and they are the best people to find them.

AI Can Make Science Feel Better Understood Than It Is

Lisa Messeri and M. J. Crockett, writing in Nature, argue that AI tools can exploit human cognitive limits and create illusions of understanding, in which people believe they understand more about the world than they do. They warn that such illusions can hide the formation of scientific monocultures, where some methods, questions, and viewpoints crowd out others, and that AI risks a phase of science in which “we produce more but understand less.”⁷

That concern is not abstract for a pharma research organization. A team that accepts AI literature summaries without reading primary sources can converge on the same set of well-cited papers, the same framing of a mechanism, and the same blind spots. Good review practice protects scientific judgment as well as accuracy.

Show the Sources and the Reasoning Behind Every Answer

The first and most basic practice is to make every answer checkable. A scientist cannot calibrate trust in a block of fluent text. They can calibrate trust in a claim tied to a specific source, a specific record, or a specific calculation that they can open and inspect.

Citations Are Necessary, and They Still Fail

Showing sources is not the same as showing correct sources. Walters and Wilder asked ChatGPT-3.5 and ChatGPT-4 to write short literature reviews on 42 topics and checked all 636 citations in the 84 resulting papers. They found that 55% of the GPT-3.5 citations and 18% of the GPT-4 citations were fabricated. Of the real citations, 43% from GPT-3.5 and 24% from GPT-4 contained substantive citation errors.⁸

Retrieval-based tools that search a defined set of documents do better, but they do not remove the problem. Nelson Liu, Tianyi Zhang, and Percy Liang audited four generative search engines and found that, on average, only 51.5% of generated sentences were fully supported by their citations, and only 74.5% of citations supported the sentence they were attached to. The authors noted that the responses were fluent and appeared informative, which made the gaps harder to see.⁹

The lesson for the lab is that a citation is a claim to be checked, not proof. Tools should make that check fast: one click from the sentence to the exact passage, page, figure, or ELN entry it came from.

What a Checkable Answer Looks Like

Teams that have done this well tend to require four things in every AI answer used for scientific work. These requirements apply whether the tool summarizes literature, searches internal reports, interprets instrument data, or drafts a protocol section.

Why Explanations Alone Can Make Things Worse

There is a trap here. It seems natural that more explanation leads to better judgment. The research says it often does not.

Gagan Bansal and colleagues ran user studies on three datasets where the AI’s accuracy was comparable to the humans’. Human and AI teams did better than either alone, but explanations did not add to that gain. Instead, explanations increased the chance that people would accept the AI’s recommendation, regardless of whether it was correct.¹⁰

Zana Buçinca, Maja Malaya, and Krzysztof Gajos tested a response to this problem. In an experiment with 199 participants, they compared three “cognitive forcing” designs (interventions that push people to think before accepting a suggestion) with two simple explainable AI designs and a no-AI baseline. Cognitive forcing significantly reduced overreliance compared with the simple explanation designs. There was a trade-off: participants gave the least favorable ratings to the designs that reduced overreliance the most.¹¹

In a lab, cognitive forcing does not need to be elaborate. Examples that work with scientific users include:

• Hypothesis first. For interpretation tasks, the scientist records their own read of the data before the AI suggestion appears.
• Open one source. Before an AI literature summary is pasted into a report, the scientist opens at least one cited passage for each key claim and confirms it says what the summary says.
• Name the alternative. For a recommendation (a compound to deprioritize, a hit to advance), the scientist writes one sentence on why the alternative was not chosen.
• Mark what was checked. The record shows which parts of the AI output the scientist verified and which they accepted without checking.

Expect some resistance. The Buçinca findings suggest that the designs people like least can be the ones that protect them most. Leaders should explain why the friction is there, keep it small, and remove steps that turn out not to catch anything.

Explainability Matters More for Prediction Models

For machine learning models used in discovery, such as property predictors, image classifiers, or generative chemistry tools, source links are less relevant than knowing what the model based its output on. José Jiménez-Luna, Francesca Grisoni, and Gisbert Schneider’s review of explainable AI in drug discovery notes that, despite a growing number of successful prospective applications, the underlying models often remain hard for people to interpret, which drives demand for explainable methods in the field.¹²

The practical step for R&D leaders is to ask, for every prediction model scientists are expected to use: can a chemist or biologist see which features, substructures, or image regions drove this prediction, and can they see how similar this input is to the training data? A prediction for a compound far outside the model’s training set should look different on screen from one for a close analog of well-characterized compounds.

Let Scientists Build the Test Set and Set the Acceptance Criteria

The second practice changes who owns the evidence. In most AI rollouts, a data science team or a vendor builds the evaluation set, runs it, and presents the results. Scientists are asked to accept a number produced by someone else, on questions someone else chose. That is the opposite of how scientists come to trust any other method.

Why Ownership Changes Trust

Berkeley Dietvorst, Joseph Simmons, and Cade Massey studied what they called algorithm aversion: people tend not to use evidence-based algorithms, especially after learning they are imperfect, even when those algorithms outperform human forecasters. In incentivized forecasting experiments, they found that people were considerably more likely to use an algorithm, and performed better as a result, when they could modify its forecasts. This held even when the modifications allowed were severely restricted. Giving people that freedom also made them more satisfied with the process, more tolerant of errors, and more likely to believe the algorithm was superior.¹³

Writing the test is a form of control. When scientists choose the questions, define what a correct answer looks like, and decide the threshold for use, they are no longer being asked to trust someone else’s claim. They are reading the results of their own experiment.

What Scientists Should Put in the Test Set

A test set built by scientists looks different from one built by data scientists. It has fewer easy cases and more of the questions that matter to the bench. Good test sets for R&D use cases tend to include:

• Known answers from the team’s own work. Questions where the lab already knows the answer from past experiments, internal reports, or published results it trusts.
• Hard cases on purpose. Conflicting literature, species differences, assay artifacts, compounds near an activity cliff, images with known staining problems.
• Questions with no good answer. Cases where the right response is “the evidence does not say” or “no internal data exists.” A tool that always produces an answer will fail these, and scientists need to see that.
• Edge-of-scope cases. Questions just outside the tool’s intended use, to show where the jagged frontier described by Dell’Acqua and colleagues falls for this team’s work.⁶
• Recent material. Questions about results or papers published after the tool’s knowledge or index cutoff.

Acceptance Criteria Written in the Lab’s Own Terms

Acceptance criteria should be written by the people who will live with the errors. A data science team might accept a summarizer with 90% agreement. A pharmacology group might decide that any error that reverses the direction of an effect, or confuses an in vitro result with an in vivo one, is disqualifying no matter how rare, while minor phrasing differences do not count at all.

The thresholds in this table are illustrations, not standards. The point is that the team that will use the tool decides them, writes them down before the test runs, and does not change them after seeing the results. That last rule is ordinary scientific discipline, and scientists will notice if it is broken.

Different Tasks, Different Verdicts

One output of a scientist-built test set should be a short list that says, in plain words, what the tool is approved for and what it is not. For example: “Approved for first-pass literature screening on well-studied targets. Not approved for summarizing safety signals. Not approved for questions about results after the index date.” That list does more for calibrated trust than any single score, because it tells a scientist which tasks have earned reliance and which still need full checking.

Build Prompts and Context With the People Who Know the Science

The third practice concerns what the AI knows about the lab’s work before it answers. Many failures that scientists see in early AI use are not failures of the model. They are failures of context. The tool did not know that “IC50” in this group’s reports means a specific assay format, that a project code refers to a particular target, or that results from a retired assay version should not be compared with current ones.

Domain Context Is Scientific Knowledge

Context for an R&D AI tool includes the instructions it receives, the documents it can retrieve, the glossary it uses, and the examples it is shown. Most of that content is scientific knowledge that only domain experts hold. It cannot be written well by an IT team or a vendor alone.

Useful context elements that scientists should help write include:

• A team glossary. Project codes, internal compound naming, assay names and versions, and abbreviations that mean different things in different groups.
• Rules for comparing data. Which assay versions can be compared, which units are standard, and which historical results are known to be unreliable.
• Instructions about uncertainty. How the tool should report conflicting results, small sample sizes, or single-experiment findings.
• Source priorities. Which internal repositories are authoritative, which are drafts, and how to treat preprints versus peer-reviewed work.
• Completed examples. A few good answers written by experienced scientists that show the expected level of detail and caution.

Context Reduces Verification Load

This work does more than raise accuracy. It reduces how hard it is to check the output. Lyell and Coiera’s finding that automation bias rises with verification complexity applies here.⁴ When the tool uses the lab’s own terms, labels the assay version, and states the species and system for every claim, the scientist can check an answer in a fraction of the time. When the tool uses generic terms and leaves those details out, every answer becomes a small research project, and people stop checking.

Treat Prompts and Context as Shared, Versioned Assets

Once context is built, it should be owned and maintained like any other shared scientific resource. That means a named owner in the science organization, a version history, and a simple way for scientists to propose changes when they find a gap. A prompt or glossary entry that fixes an error should be linked to that error in the error log, so the team can see that the problem was addressed and test that the fix worked.

In regulated work downstream of research (GLP studies, GMP quality control, clinical documents), changes to prompts and context also fall under change control. In early discovery work, the formality can be lighter. The habit of recording what changed and why should be the same in both, because a discovery tool often moves into regulated use later, and the record of how it was built will be asked for.

Run AI and Scientist Side by Side Before Anyone Relies on It

The fourth practice gives scientists direct experience of the tool’s performance on their own work before any decision depends on it. A test set shows how the tool performs on chosen questions. A side-by-side period shows how it performs on the real flow of work, with all its mess.

What Clinical AI Teams Learned From Silent Trials

Clinical AI teams have a name for this: the silent trial. Jethro Kwong and colleagues at the University of Toronto and The Hospital for Sick Children describe it as a phase in which a model is evaluated on prospective patients in real time while clinicians are blinded to its predictions, so the predictions do not influence care. In their case, a model to predict obstructive hydronephrosis in infants performed poorly on the silent trial data, with its AUC (area under the curve, a standard measure of discrimination) falling from 0.90 to 0.50. The team traced the drop to shifts in patient age distribution, which kidney was affected, and imaging format. After correcting those issues, performance held across two independent silent trial datasets, with AUC between 0.85 and 0.91.¹⁴

That case makes two points. First, a model that looked strong in development failed on real incoming data, and only a prospective period showed it. Second, the problems were the kind domain experts recognize: changes in the population and the way data was produced. The silent trial also found that patients did not understand how the AI would be used in their care, which shaped the final interface design.¹⁴

How Trust Develops Through Experience

Katharine Henry and colleagues interviewed 20 clinicians (13 physicians and 7 nurses) who used a deployed machine learning system for early sepsis warning. Rather than seeing the system as a replacement for their judgment, clinicians described themselves as partnering with it. The authors concluded that clinicians can build trust with such a system through experience, expert endorsement and validation, and systems designed to respect their autonomy and support their whole workflow.¹⁵ Clinicians described coming to understand the system by watching how it behaved across different scenarios and patient types, without needing to understand the machine learning underneath. They also wanted to know whether the validation population resembled their own patients.¹⁵

Bench scientists are no different. They will trust a tool they have watched perform on their own compounds, their own assays, and their own literature questions, and they will want to know whether any external validation reflects their kind of work.

Designing a Side-by-Side Period in the Lab

Count What the AI Catches, Not Only What It Misses

Side-by-side periods are usually framed as a way to find AI errors. They are just as useful for finding human ones. If the tool flags an experiment the scientist did not find, or a paper that contradicts the team’s working hypothesis, that is evidence of value the scientists can see for themselves. Recording both directions also keeps the period from becoming a contest. The question is not whether the AI or the scientist wins. It is what the combination does better than either alone, which is the “complementary performance” Bansal and colleagues set out to measure.¹⁰

Keep a Visible Error Log and Show How Each Error Was Fixed

The fifth practice deals with what happens after an error is found. In many organizations, AI errors are reported in a chat thread or mentioned in a meeting and then forgotten. The scientist who found the error never hears what happened. The next scientist finds the same error. Trust erodes, and it erodes faster than accuracy numbers can rebuild it.

Why Hidden Errors Destroy Trust

Samir Passi and Mihaela Vorvoreanu’s review of about 60 papers on overreliance defines the problem simply: overreliance occurs when users start accepting incorrect AI outputs, which can lead to errors that ultimately make people lose trust in AI systems.¹⁶ The pattern runs in both directions. People who are not shown errors overtrust. People who see errors with no visible response undertrust, and the algorithm aversion research suggests that once people see an algorithm err, they are quick to abandon it.¹³

A visible error log addresses both problems. It shows scientists where the tool goes wrong, which supports calibrated trust. It also shows that errors lead to action, which gives scientists a reason to keep reporting them and to keep using the tool where it has earned that.

What the Log Should Contain

The log does not need special software. A shared, searchable list that every user can read is enough to start. What matters is that each entry is complete enough to learn from.

Make Reporting Easy and Credit the Reporter

Scientists will report errors when it takes a minute and when reporting is valued. Two habits help. First, build a report button or a short form directly into the tool, so reporting does not require leaving the work. Second, recognize the people who find errors. In a team meeting, name the scientist whose report led to a fix. The message is that finding errors is part of the scientific work of using AI, not a sign of disloyalty to the project.

Not Every Error Gets Fixed, and That Should Be Visible Too

Some errors come from limits of the underlying model that the team cannot change. The log should say so plainly and record the response: narrowing the approved scope, adding a required check for that question type, or warning users in the tool itself. Being honest about what cannot be fixed builds more trust than a log that only shows successes. Scientists know that every method has limits. What they distrust is a method whose limits are hidden.

Governance That Includes the Bench

The sixth practice is about who decides. In many pharma and biotech organizations, AI governance is run by IT, data science, legal, and quality, with research represented by a senior leader. That group is needed, but it is often missing the people who use the tools every day and see their failures first: the bench scientists.

Why the Bench Needs a Seat

The Henry study found that clinicians valued systems that respected their autonomy, including the ability to ask about design choices, adjust settings such as alert sensitivity, and give input to the deployment team.¹⁵ The Dietvorst findings point the same way: some control over an algorithm makes people more willing to use it and more tolerant of its errors.¹³ Governance is where that control becomes real. A scientist who helps decide what a tool is approved for will trust that decision more than one handed down from a committee of non-users.

There is also a practical reason. Bench scientists know which errors matter. They know which assay results are fragile, which literature is contested, and which shortcuts their colleagues are already taking with public AI tools. Without them, governance decisions are made on partial information.

What Cross-Functional R&D Governance Looks Like

A workable structure for AI in research usually has a small standing group with clear roles:

• Bench scientists from each major user group, rotated so the role does not fall on the same few enthusiasts.
• A science lead who owns the decision on what each tool is approved for in research use.
• The context owner who maintains prompts, glossaries, and retrieval sources.
• Data science or IT to explain what can and cannot be changed technically.
• Quality, when a tool may move into GLP, GMP, or clinical use, so that the record built in research can support later validation.

The group’s standing agenda can be short: new error log entries and patterns, test set results, requests to widen or narrow approved scope, and changes to context. Its outputs should be visible to every user: the current approved-use list, the error log summary, and what changed since last time.

Training That Changes the Work, Not Just Awareness

Parasuraman and Manzey’s finding that training and instructions alone do not prevent automation bias does not mean training is useless.³ It means training has to be tied to the working practices above. Effective training for scientists covers how to open and check a source quickly, how to record their own read before seeing the AI’s, how to report an error in a minute, and what the current approved-use list says. It also shows real entries from the team’s own error log, because nothing calibrates trust faster than seeing how the tool failed on a colleague’s question last month.

The Wiley finding that 57% of researchers see a lack of guidelines and training as a main barrier, and that only 41% feel adequately supported, suggests there is real demand for this.¹ Scientists want to know how to use these tools well. Governance that includes them is the most direct way to give them that.

A Practical Sequence for the First Ninety Days

For an R&D organization starting from informal AI use, the six practices can be introduced in a sequence that builds evidence as it goes.

The sequence is deliberately slow for the first use case. The first one sets the pattern. Once scientists have seen a tool tested on their questions, compared against their own work, and corrected when they reported errors, the next use case moves faster because the habits and the trust already exist.

Conclusion

Scientist trust in AI is not a matter of persuasion. The research is consistent: people working with automated aids, experts included, accept wrong suggestions when checking is hard, explanations can raise acceptance of wrong answers, and training on its own does not fix it. What does help is a set of working practices that make AI output checkable, put scientists in charge of the evidence, give them direct experience of the tool on their own work, and show them that errors lead to action. Those practices aim at appropriate reliance, where scientists lean on the tool where it has earned that and check it hard where it has not. They also protect something an accuracy number cannot measure: the team’s own scientific judgment.

Sakara Digital works with pharma and biotech organizations putting AI into research and quality work in a way scientists will use and inspectors will accept. If you are trying to move from scattered individual AI use to practices your scientists trust, and want an independent perspective on where to start, we are happy to have that conversation.

For Further Reading