Why AI Adoption in Quality Stalls After the Pilot

What a Stalled Quality Pilot Looks Like

Anyone who has run digital projects in a quality organization will recognize the pattern. A small team picks a use case that everyone agrees is painful: deviation trending, complaint triage, first-pass review of batch records, drafting CAPA effectiveness summaries, or searching SOPs for the answer to an inspector’s question. A vendor or an internal data team builds something in eight to twelve weeks. The results on the test set look good. The steering committee is pleased. Then the project goes quiet.

The Pilot Worked. Then Nothing Happened.

When you look closely at a stalled pilot, the story is almost never “the AI did not work.” It is one of these:

• The validation lead was brought in after the pilot ended and asked for requirements, a risk assessment, and test evidence that nobody had kept. The team is now rebuilding the pilot as a validation project, and that is a six-month conversation.
• IT owns the tool, the data team owns the model, and the quality function that uses the output does not own either. When the vendor asks who will approve a model update, there is no answer.
• The pilot ran on a cleaned extract of two years of deviation records. Production data from three sites uses different category codes, free-text fields in two languages, and a legacy system nobody wants to touch.
• Ten enthusiastic users took part. The two hundred people who would use the tool in production have never seen it, and their managers have not been asked how it changes their review steps.
• The pilot report says the tool was “accurate and well received.” Nobody wrote down before the pilot what result would justify rollout, so the budget discussion turns into a debate about opinions.

None of these is a technology problem, and none of them is new. What is new is how often they appear together, and how much harder they are to fix after a pilot than before one.

What the Research Says About the Drop After the Pilot

Cross-industry research tells a consistent story: moving AI from pilot to production is where most of the loss happens, and the reasons are mostly organizational. The MIT NANDA report on generative AI in business found that 60 percent of organizations had evaluated enterprise-grade, task-specific AI tools, only 20 percent had reached the pilot stage, and just 5 percent had reached production.¹ The authors are open that these are preliminary findings based on 52 interviews, 153 survey responses, and a review of more than 300 public AI initiatives, and that the figures come from interviews rather than official company reporting. Even read with that caution, the size of the drop is hard to ignore.

Deloitte’s fourth-quarter 2024 survey of generative AI in the enterprise found the same shape from a different angle. Over two-thirds of respondents said that 30 percent or fewer of their current experiments would be fully scaled in the next three to six months.² Deloitte also reported that regulatory compliance had become the top barrier holding organizations back from developing and deploying generative AI tools.²

The reasons given for the drop are also consistent. RAND interviewed 65 experienced data scientists and engineers and identified five root causes of AI project failure. The first was that stakeholders misunderstand or miscommunicate the problem to be solved, so that models end up “optimized for the wrong metrics” or do not fit the business workflow.⁴ The others were a lack of the data needed, a focus on technology over the problem, inadequate infrastructure to manage data and deploy models, and applying AI to problems that are too hard for it.⁴ Only the last of these is a limit of the technology itself.

Prosci’s research reached a similar conclusion from the change management side. In its study of 1,107 professionals, 63 percent of AI implementation challenges came from human factors, user proficiency was the single largest category at 38 percent, and technical implementation issues accounted for 16 percent.³

Why Quality Is Harder Than Most Functions

Everything in the general research applies to quality, and quality adds three burdens of its own.

First, the output of a quality AI tool often feeds a GxP decision: whether a deviation is critical, whether a batch record entry needs follow-up, whether a complaint is reportable. That brings validation, data integrity, and change control expectations into play the moment the tool leaves the sandbox.

Second, the regulatory position for AI in GMP settings is still forming. A 2026 industry survey on digital and AI-enabled tools in pharmaceutical development and manufacturing, run by the Digital CMC Centre of Excellence in Regulatory Science and Innovation with support from the UK MHRA, found that the main challenges were culture change, skills gaps, and uncertainty about regulatory expectations, particularly for AI-enabled applications and the evidence needed for submissions and inspections.⁵ Fewer than 15 percent of the tools respondents described had been included in submissions.⁵

Third, quality teams are small, busy, and rightly cautious. The people who would own an AI-assisted process are the same people running investigations, hosting audits, and closing CAPAs. If a pilot asks them for time after it ends, it is competing with work that has a regulatory due date.

An ISPE article on implementing AI in regulated pharmaceutical settings put the point plainly: organizations are not short of AI ideas, and the hard part is moving beyond promising pilots into compliant, production-ready systems.⁶ The same piece noted that while many organizations launch pilot projects successfully, far fewer have clear plans for sustaining performance, documenting model changes, and demonstrating ongoing control.⁶ That gap between the pilot and the plan is the subject of the rest of this article.

Pilot Conditions Are Not Production Conditions

A pilot is designed to remove obstacles so that a team can find out quickly whether an idea works. That is the right design for learning. It is the wrong design for scaling, because the obstacles a pilot removes are exactly the ones production will put back.

The table below sets out the most common differences we see between how a quality AI pilot is run and what production in a GxP setting requires. Every row on the right is a piece of work. If that work is not planned during the pilot, it becomes the reason the pilot stalls.

None of these production requirements is unreasonable, and none of them is specific to AI. They are the ordinary expectations for any computerized system that supports a GxP process. The difference with AI is that pilots are often run outside the usual project governance, because they are small, fast, and labeled experimental. That label is useful for speed. It becomes a problem when the pilot succeeds and the organization discovers it has to start the regulated work from the beginning.

Stall Point One: The Validation Path Was Never Decided

The most common reason a successful quality AI pilot stalls is that nobody decided, at the start, how the tool would be validated if it worked. The pilot team assumed validation would come later. Validation, when it arrives, asks for things the pilot did not produce.

Why Pilots Skip This Question

There are understandable reasons. Validation leads are busy and are often brought in only for systems that are definitely going live. The pilot team worries that involving validation early will slow things down. And for newer AI techniques, it is not always obvious what “validated” should mean, so it feels easier to wait until the answer is clearer.

That last point is real. The regulatory position on AI in GMP settings is still developing. A 2026 IFPMA position paper on AI in pharmaceutical manufacturing observed that most national regulatory authorities have so far published reflection papers and draft guidance in this area.⁹ In the EU, Annex 22 on artificial intelligence was published in 2025 as a draft supplementary guideline to the EU GMP Guide.⁸ As of this writing it remains a draft, with no final text. IFPMA described the draft as the first document to introduce specific exclusions, such as generative AI or dynamic models for high-risk GMP applications, and noted that it links to the computerized systems guidance in Annex 11 and the documentation guidance in Chapter 4.⁹ In the US, IFPMA lists FDA’s 2025 draft guidance on using AI to support regulatory decision-making for drugs and biological products, alongside CDER’s 2023 discussion paper on AI in drug manufacturing.⁹

Uncertainty about the final rules is a reason to decide the validation approach early, not late. The questions that matter most do not depend on the final wording of any guidance. They depend on what the tool does and how much its output affects a GxP decision.

What to Settle Before the Pilot Starts

A short validation planning session before the pilot, with the validation lead, the future process owner, and the technical lead in the room, should answer the following:

1. Intended use. What exactly will the tool do in production, and what will it not do? “Suggests a deviation category for the investigator to confirm” is an intended use. “Helps with deviations” is not.
2. GxP impact and risk. Does the output inform a decision about product quality, patient safety, or data integrity? If a wrong output were accepted without challenge, what is the worst plausible result? This drives how much testing and control is needed.
3. Model type. Is the model static once released, or does it change as it sees new data? Is it generative? Given how the draft Annex 22 treats dynamic and generative models in high-risk GMP applications, this choice can decide whether a use case is feasible in its planned form.⁹
4. The human step. Where does a qualified person review, accept, or override the output, and how is that recorded?
5. Evidence to keep. Which pilot outputs will be reused as validation evidence: the test data set, the acceptance criteria, the error analysis, the configuration record? If they will be reused, they need to be controlled from the start.
6. Change control after release. Who decides whether a retrained model, a new prompt, or a vendor version update is a minor or major change?

With these answers in hand, the pilot can be designed as the first stage of the validation lifecycle. The test set becomes a controlled record. The acceptance criteria are written before the results are known. The error analysis is documented in a form that a validation report can reference. None of this makes the pilot much slower. It does make the pilot far more useful when it succeeds.

When the Answer Is “Not Yet”

Sometimes the validation planning session shows that the intended use is not feasible in a GxP setting today, for example a generative model making a determination that feeds directly into batch release. That is a valuable result. It means the pilot can be redesigned toward a lower-risk intended use (drafting for human review, or supporting a non-GxP step) or stopped before anyone spends money on it. Finding this out in week one is far better than finding it out in month six.

For a deeper treatment of how to scale validation effort to risk for AI systems, see our guide to risk-based AI validation in GxP environments.

Stall Point Two: No One in Quality Owns the Process

The second stall point is ownership. In many AI pilots, the sponsor is a digital or innovation leader, the builder is a data team or vendor, and the quality function is the customer. That arrangement is fine for a pilot. It does not work for a GxP process in production, because production needs someone who can make decisions on behalf of quality and be accountable for them.

IT Owns the Tool, Nobody Owns the Outcome

Quality professionals know the distinction between a system owner and a process owner. The system owner is responsible for the computerized system: availability, security, backups, and technical changes. The process owner is responsible for the business process the system supports: how it should work, what the outputs mean, and whether the process is under control. In a traditional QMS rollout, the process owner is usually obvious. The head of deviation management owns the deviation process, whatever system it runs in.

AI pilots blur this because the AI tool often changes the process itself. If a model suggests deviation categories, someone has to decide whether investigators must accept, confirm, or ignore the suggestion; whether the suggestion is part of the record; and what happens when the suggestion is wrong. Those are process decisions, not system decisions. If the pilot was run by IT or a data team, those decisions have often been made informally by the people who built the tool.

The MIT NANDA research found that the most effective AI-buying organizations drive adoption through distributed experimentation, vendor partnerships, and clear accountability, and concluded that companies succeed when they decentralize implementation authority but retain accountability.¹ In quality, accountability has a specific meaning. It means there is a named person who will stand in front of an inspector and explain how the AI-assisted process works and why it is under control.

What the Process Owner Decides

The future process owner should be named before the pilot starts and should take part in it, not just receive the results. Their decisions include:

• Which parts of the current SOP change, and how the human review step is written.
• What error rate is acceptable, and for which types of error. A missed critical deviation is not the same as a mislabeled minor one.
• What the reviewer must record when they accept or reject an AI output.
• Who can approve a change to the model, prompt, or configuration, and what testing is needed first.
• What monitoring data they want to see, how often, and what triggers a review.
• When the tool should be switched off, and what the fallback process is.

These decisions become the main content of the production SOP and the change control plan. If the process owner makes them during the pilot, they can be tested during the pilot. If they are left until afterward, the process owner is being asked to approve a process they did not design, and a cautious owner will say no or ask for time. That is a stall.

Why Ownership Gets Skipped

The usual reason is capacity. Quality leaders are asked to “stay close” to an AI pilot on top of a full workload, and there is no time set aside for it. The fix is simple to describe and harder to do: agree before the pilot how many hours a week the process owner and one or two reviewers will spend on it, and get their manager’s commitment to protect that time. If the organization cannot find that time for the pilot, it will not find it for the rollout, and it is better to know that at the start.

IFPMA’s paper made a related point about human oversight: to manage the risk that people over-trust AI outputs, it is important that the human in the loop is trained and qualified and that the responsibility is clearly built into the role.⁹ Built into the role means it appears in the job description, the SOP, and the training curriculum, not only in a project plan.

Stall Point Three: The Data Was Ready for the Pilot, Not for Production

The third stall point is data. Almost every quality AI pilot runs on data that has been prepared for it. Someone exports two or three years of records, removes duplicates, fixes the obvious errors, maps the categories to a common list, and hands over a tidy file. The model performs well on that file. Then the team tries to connect the tool to live data and discovers why the file needed so much work.

The Curated Data Set Problem

Quality data has particular features that make this gap wide:

• Several systems and sites. Deviations, CAPAs, complaints, and change controls are often kept in different systems, and each site may have configured the same system differently.
• Free text. Much of the useful information in a quality record is written by people in narrative fields, with site-specific terms, abbreviations, and sometimes more than one language.
• Category drift. Classification lists change over time. A root cause category that existed in 2021 may have been split into three in 2024.
• Legacy records. Older records migrated from paper or earlier systems may be incomplete or structured differently.
• Controlled access. Some records contain personal or commercially sensitive information, which limits what can be copied into a new environment.

This is not unique to quality. An ISPE article on scaling AI in pharma manufacturing listed data readiness and data silos as the first obstacle to scaling, describing fragmented, inconsistent, and unstructured data across systems, and said that breaking down silos and getting high-quality, unified, contextualized data is the larger share of the work of AI readiness.⁷ A separate ISPE piece on AI in quality named data availability and integrity as the first of its implementation challenges, ahead of model validation, change management, and security.¹⁰

Data Integrity Applies to the Whole Pipeline

In a GxP setting, data readiness is not only about whether the model can use the data. It is also about whether the data flowing into and out of the model meets data integrity expectations. A 2025 review of regulatory perspectives on AI in GMP settings made the point that AI and machine learning systems must uphold ALCOA+ principles across the whole data pipeline, including training, testing, and deployment.¹¹ ALCOA+ is the set of data integrity attributes regulators expect: attributable, legible, contemporaneous, original, accurate, complete, consistent, enduring, and available.

For a quality AI tool, this raises practical questions the pilot often does not answer. Can you show which version of the source data the model used for a given output? If an investigator relies on an AI-generated summary, is the summary kept as part of the record, and is it attributable? If the source record is later corrected, what happens to the AI output that was based on the old version?

Visibility of data sources is also a current problem across the industry. A Pistoia Alliance poll of life sciences professionals at its 2025 US conference found that more than 1 in 4 (27 percent) did not know what scientific content their organization’s AI or LLM systems used, or relied only on titles and abstracts.¹² That poll was about R&D content rather than quality records, but the lesson carries over: if the people using the tool cannot say what it draws on, they cannot judge whether to trust it.

The Production Data Test

The fix is to test the model on production-shaped data during the pilot, not after it. That does not mean connecting to every live system on day one. It means that before the pilot starts, the team agrees on a data readiness check covering four points.

If this check shows that production data is far from ready, the pilot does not have to stop. It may be better to narrow the scope to one site with good data, or to run a data remediation effort in parallel. Our article on prioritizing data cleanup for AI readiness covers how to decide where to start. What matters is that the decision is made knowingly, before the pilot results are presented as a success that can be rolled out.

Stall Point Four: Users Were Shown the Tool, Not Trained for the Role

The fourth stall point is training, and it is often mistaken for a communication problem. A pilot with ten volunteer users produces enthusiastic feedback. The rollout plan includes a demo and a quick reference guide. Then the wider group of reviewers either does not use the tool, uses it without checking its outputs, or uses it inconsistently. The first outcome wastes the investment. The second is a compliance risk. The third makes the process hard to control.

Knowing the Tool Is Not the Same as Being Able to Use It

Prosci’s research found that user proficiency was the largest single category of AI implementation challenges, at 38 percent, made up of learning curve difficulties (22 percent), problems with prompting (11 percent), and inadequate training (6 percent).³ Prosci’s analysis draws a useful line between knowing how a tool works and being able to use it well under real working conditions, and argues that the second needs practice, coaching, and time rather than a single training session.³

In life sciences specifically, skills are becoming a larger barrier, not a smaller one. The Pistoia Alliance’s 2025 Lab of the Future survey of more than 200 experts found that 34 percent cited a lack of people as a barrier to AI adoption, up from 23 percent in 2024.¹³ Respondents asked for best practice guides (51 percent), AI and machine learning courses (45 percent), and skills training (40 percent).¹³ The 2026 industry survey on digital CMC tools also named skills gaps among the principal challenges, and found strong support for more training.⁵

What Quality Reviewers Need to Be Able to Do

For a quality AI tool, training has to go beyond “how to click through the screen.” A reviewer working with AI output needs to be able to:

• Explain in plain terms what the tool does and what it does not do, in line with its intended use.
• Recognize the types of error the tool is known to make, based on the pilot’s error analysis.
• Check an output against the source record efficiently, and know when a closer look is needed.
• Reject or correct an output and record why, in the way the SOP requires.
• Report a suspected problem with the tool through the right channel, such as a deviation or an incident.

The third and fourth points address the risk that concerns regulators most in human-in-the-loop designs. IFPMA’s paper describes automation bias, the tendency to accept automated results without a thorough check, as an inherent risk that may be more likely with the human-like outputs of generative AI, and says that the human in the loop must be trained and qualified.⁹ A reviewer who has only seen the tool succeed during a demo is poorly prepared to catch the cases where it fails.

Use the Pilot to Build the Training

The pilot is the best source of training material the organization will ever have. It contains real examples of correct outputs, subtle errors, and obvious failures, all on the organization’s own records. If the pilot is planned with training in mind, the team can:

• Collect a set of worked examples, including the model’s mistakes, for use in training and competence checks.
• Record the questions pilot users asked, which become the FAQ for the wider rollout.
• Have pilot users help design the review step in the SOP, so it reflects how the work is done.
• Identify two or three pilot users who can coach colleagues during the rollout.

Training for AI-assisted roles also needs to be recorded in a way that shows competence, not just attendance. We cover how to build role-based quality training for AI oversight in more depth in our six-month curriculum for reskilling quality teams.

Stall Point Five: No One Agreed What Success Would Mean

The fifth stall point is the one that turns the other four from delays into stopped projects. If nobody agreed before the pilot what result would justify going to production, then the end-of-pilot review becomes a negotiation. Supporters point to positive feedback. Skeptics point to the validation work and data effort still ahead. Budget holders ask what the return will be, and nobody has a number they trust. The safest decision in that meeting is to wait, and waiting is how pilots stall.

Accuracy Is Not a Business Result

Most pilot reports lead with a model performance figure: accuracy, agreement with human reviewers, or precision and recall. Those figures matter for validation. They do not answer the question the budget holder is asking, which is whether the quality process is better as a result.

RAND’s finding that models are often “optimized for the wrong metrics” applies here directly.⁴ A deviation classifier can be highly accurate on categories that investigators rarely disagree about and still leave the difficult cases, the ones that take the most time, untouched. A pilot that measures only overall accuracy will miss that.

It is worth noting how the MIT NANDA researchers defined success in their own study: deployment beyond the pilot phase with measurable KPIs.¹ In other words, a pilot is not a success because it performed well. It is a success when it reaches production and moves a measure the organization cares about.

Measure the Quality Outcome, Not the Activity

This article deliberately sets aside measures of activity, such as how many people have access to the tool or how often it is opened. Those have their place in tracking a rollout, but they do not tell a quality leader whether the process improved. The question here is narrower: what change in the quality process would justify the effort of validating, training, and supporting this tool in production?

The answer depends on the use case. The table below gives examples of outcome measures for common quality AI use cases. The important step is to choose one or two before the pilot, take a baseline from current performance, and write down what change would justify rollout.

Set the Decision Rule in Advance

Once the measure and baseline are agreed, the steering group should also agree a decision rule before the pilot starts. For example: “If reviewer time per investigation report falls by at least a quarter, with no increase in QA review cycles, and the validation plan and data readiness check show no blocking issues, we will fund the rollout to the two remaining sites.” The specific threshold is the organization’s choice. What matters is that it is written down, dated, and signed by the people who will make the funding decision.

A decision rule also makes it easier to stop a pilot that is not working, which is its own kind of success. Our article on setting exit criteria on day one covers the other side of this decision in more detail.

A Scale-Ready Pilot Charter: What to Settle Before Day One

The five stall points share one feature: each is much easier to handle before the pilot starts than after it ends. The practical tool for this is a short pilot charter, agreed by the sponsor, the future process owner, the validation lead, IT, and the data or vendor team before any build work begins. It does not need to be long. Three to five pages is enough for most quality use cases.

The sequence below is how we suggest building it. Each step produces a section of the charter.

Questions the Charter Should Answer

If you want a quick way to test a draft charter, check that it gives a clear answer to each of these questions. A charter that cannot answer them is not yet ready to approve.

Rescuing a Pilot That Has Already Stalled

Many readers will already have a pilot that finished well and has not moved. The charter can be used after the fact as a diagnostic. Take the five questions in the table above and answer them honestly for the stalled pilot. In our experience, the gaps usually fall into one of three patterns.

• One or two gaps, both fixable. For example, no named process owner and no baseline for the outcome measure. Name the owner, reconstruct the baseline from historical data if possible, and run a short second phase designed to fill the gaps. This is the most common case and often the quickest to resolve.
• The validation path is blocked. The intended use turns out to be too high-risk for the model type, or the draft regulatory position makes it uncertain. Consider whether a lower-risk intended use would still deliver most of the value, such as drafting for human review instead of making a determination.
• The value case does not hold. Once a baseline is taken, the improvement is small compared with the effort of validation, data work, and training. Stop the pilot, document what was learned, and move the team to a use case with a stronger case.

What we advise against is running a second, larger pilot without fixing the gaps. A bigger pilot with the same missing pieces will stall in the same place, with more people and more money involved. Our earlier article on moving from AI pilot to enterprise scale in regulated environments covers the enterprise side of scaling once a use case is ready.

Conclusion

AI adoption in quality stalls after the pilot for reasons that are predictable and, for the most part, preventable. The validation path is left undecided. No one in quality owns the process. The data is prepared for the pilot but not for production. Users are shown the tool but not trained for their role in reviewing it. And nobody agrees in advance what result would justify the effort of going live. The research from MIT NANDA, Deloitte, RAND, and Prosci points the same way: the drop from pilot to production is mostly about organization and planning, and the life sciences surveys add regulatory uncertainty and skills gaps to that list. As the MIT NANDA authors put it, “The core barrier to scaling is not infrastructure, regulation, or talent. It is learning.”¹ In a GxP setting we would put it slightly differently. Regulation is part of the problem, and the organizations that learn fastest are the ones that plan for it from the start. The fix is not a more elaborate pilot. It is a short charter, agreed before day one, that answers five questions and turns each pilot output into the first draft of what production will need.

Sakara Digital works with pharma and biotech quality, IT, and data leaders who want their AI pilots to reach production in a state they can defend to an inspector. If you have a pilot that has stalled, or you are about to start one and want an independent view on the charter, validation approach, or data readiness before you begin, we are happy to have that conversation.

For Further Reading