The Manufacturing Data Quality Scorecard: KPIs Beyond Regulatory Submissions

Why ALCOA+ Alone Is Not a Manufacturing Scorecard

Talk to almost any pharma quality leader about manufacturing data quality and the conversation converges on the same vocabulary. ALCOA+ (attributable, legible, contemporaneous, original, accurate, plus complete, consistent, enduring, available). 21 CFR Part 11 audit trails. Annex 11 controls. GxP validation. These are the pillars of data integrity, and they matter. But they are compliance vocabulary. They tell you whether an inspector will find a finding. They do not tell you whether the manufacturing data that finance, supply chain, digital, and R&D consume every day is fit for purpose.

That gap becomes obvious the moment you sit through a management review. The head of quality shows a bar chart of open deviations. The site head shows batch yield. Someone from IT talks about system availability. Nobody in the room can answer a simple question: how confident are we in the data that supports our next annual product review, our next control-strategy change, or the digital twin we are quietly relying on for scheduling?

The FDA’s Quality Management Maturity (QMM) program is one signal that this conversation is shifting. QMM’s 2024 pilot assessed nine manufacturing establishments on management commitment, technical excellence, and advanced quality systems, well beyond baseline CGMP compliance, and the program continues to evolve in its third year of voluntary assessments as of 2026.¹ The direction of travel is clear: regulators want to see that manufacturers are running quality on evidence, not on checklists.

The three failure modes ALCOA+ misses

An ALCOA+ compliant record can still fail in ways that matter operationally. We see three consistent failure modes when we audit manufacturing data programs:

• Reconciliation gaps. The MES batch record and the LIMS result set match individually against Part 11 requirements but disagree with each other on quantities, timestamps, or sample identity. Both records are ALCOA+ compliant. The site cannot answer which one is right.
• Model input rot. The digital twin that is meant to predict yield is trained on data that is technically clean, but drawn from a period when a critical sensor was drifting. The twin is now confidently wrong.
• Trend blindness. Individual OOS investigations are documented. Trend analysis across investigations is not, or is done in a spreadsheet that nobody quality-controls. The FDA has flagged this pattern repeatedly, with failure to investigate discrepancies cited in 50 of 148 recent lab-focused warning letters.²

None of these show up on a submissions readiness scorecard. All of them show up in inspections, batch failures, and eventually in warning letters. A manufacturing data quality scorecard has to make them visible before that happens.

The consequence of measuring the wrong things

When leadership measures the wrong things for long enough, two consequences follow. First, the quality organization gets very good at producing reports that pass an inspection but do not change how the plant runs. The reports become an artifact rather than a management tool. Second, when a real problem does surface (a repeat OOS, an inspector question about a digital tool, a supply chain disruption traceable to a batch record disagreement) the leadership team discovers that the data infrastructure is not ready to help them investigate. They have measured compliance for years while the underlying data debt quietly accumulated.

This is why FDA’s Quality Management Maturity assessment protocol evaluates management commitment, business continuity, technical excellence, advanced quality systems, and employee engagement, rather than a checklist of ALCOA+ conformance.¹ The regulator is trying to encourage the industry to measure the things that predict operational reliability, not the things that describe past compliance. A manufacturing data quality scorecard should follow the same principle.

Six Data Quality KPIs Beyond Regulatory Submissions

The scorecard we propose is built around six KPIs. Each measures something distinct about the trustworthiness of manufacturing data. Each can be defined operationally, tracked monthly, and rolled up to a single site score. And each maps to a decision that someone on the leadership team is already making, whether they realize the data behind it is instrumented or not.

The remaining sections define each of these in enough operational detail that a quality director can hand the definitions to a systems and data team and get to a working measurement within a quarter.

KPI 1 & 2: Process Capability and Deviation Rate Reliability

KPI 1: Process capability, measured from raw batch data

Cpk and Ppk are the workhorses of pharma process performance. Cpk measures potential capability under ideal conditions; Ppk measures actual performance across all sources of batch-to-batch variation.³ Most manufacturers report these once a year in the annual product review, where a Cpk trend is expected to be shown over three years alongside deviation rates and complaint counts.⁴

The industry benchmarks are well established: Cpk of at least 1.33 is the general standard for a capable process, and 1.67 or higher is considered world-class in pharma.³ A sample of at least 30 batches is typically recommended to make the estimate statistically meaningful. Notably, the FDA does not prescribe a universal numeric threshold; sponsors are expected to justify how these metrics are used within their own validation and monitoring framework.³

The scorecard-relevant question is not “what is our Cpk?” It is “is our Cpk being calculated on data we would defend?” That means:

• Cpk is calculated directly from batch execution records in MES, not from a hand-curated APR spreadsheet
• The critical quality attribute list, and the associated specifications, are governed and version-controlled
• Cpk trends are refreshed at least monthly, not annually, so that a declining trend can act as a leading indicator of a future failure rather than a lagging obituary
• The Cpk calculation methodology is documented in a way that a QA reviewer, an internal auditor, and an FDA investigator would all reach the same number

KPI 2: Deviation rate reliability

Deviation rate per batch is one of the most heavily reported quality metrics, tracked in FDA quality metrics discussions and in every APR.⁵ Increasing deviation rates signal control loss; decreasing rates indicate improvement. The FDA also specifically tracks repeat deviation rate as part of pharmaceutical quality system effectiveness.⁵

But rate reliability is a different problem than rate improvement. Rate reliability asks:

• How consistently are deviations being identified? A site with a suddenly falling deviation rate may not have improved. It may have started under-reporting.
• How consistently are deviations being coded? If one shift codes a wrong-material event as “documentation error” and another codes it as “material handling”, the trending is worthless.
• How consistently are they being closed? Deviations that sit open for six months are almost always a sign that the underlying investigation data was inadequate.

Practically, deviation rate reliability can be operationalized as a composite of three sub-indicators. Percent of deviations closed within target cycle time. Percent of deviations coded to a controlled taxonomy. And percent of deviations with linked evidence in the connected systems (MES, LIMS, eQMS). The FDA’s own view is that CAPA effectiveness, audit compliance, and documentation accuracy are among the most critical KPIs, alongside batch rejection rate and OOS rate.⁶ Deviation rate reliability essentially instruments the data quality that makes those other metrics believable.

There is also a governance angle to deviation rate reliability that quality organizations often underestimate. A deviation taxonomy that changes twice a year, or that has parallel local variants at each site, produces a rate that cannot be trended across time or across the network. The scorecard should not just measure the current-period conformance to a taxonomy. It should also measure how stable the taxonomy itself has been over the trailing 24 months. A rapidly changing taxonomy is a signal that the data is not yet trustworthy enough to compare year over year, no matter how compliant any individual record is.

KPI 3 & 4: Lab OOS Trend Accuracy and Real-Time Release Readiness

KPI 3: Lab OOS trend accuracy

Out-of-specification results are unavoidable. What matters is whether the site can trend them correctly and act on the trend. EU regulations explicitly require laboratories to trend their critical data and investigate OOS results, and the FDA has repeatedly criticized companies for failing to conduct scientifically sound and conclusive investigations into repeat OOS assay and stability results.⁷

The recurring failure mode in warning letters is not that no investigation happened. It is that investigations happened in a way that broke the trend. Multiple OOS results were invalidated as “laboratory error” without specific error identification. Recurring use of vague “analyst error” without documented mistakes. Statistical tests used alone to reject OOS results. Each of these makes any subsequent trend analysis meaningless.⁸

Lab OOS trend accuracy, as a scorecard KPI, is the percentage of OOS results in the reporting period that meet all of the following criteria:

• Investigation completed within the site’s target cycle time
• Root cause coded to a controlled taxonomy that the trending system understands
• Not invalidated on the basis of “analyst error” without a specific documented error
• Linked to the batch, instrument, method, and analyst in the underlying LIMS data
• Included in the site’s current trend analysis, not archived out of it

KPI 4: Real-time release testing readiness

Real-time release testing (RTRT) is one of the most consequential capabilities in modern pharma manufacturing. It replaces some or all traditional end-of-batch laboratory testing with in-process data collected by PAT sensors and process models. ICH Q13 explicitly emphasizes RTRT as central to continuous manufacturing, and the FDA’s January 2025 guidance update on 21 CFR 211.110 explicitly supports advanced technologies, real-time quality monitoring, PAT, and continuous manufacturing systems.⁹

Even manufacturers with no near-term intent to file for RTRT should be measuring their readiness, because the readiness score is a direct proxy for the maturity of their manufacturing data infrastructure. If you cannot in principle release a critical quality attribute on process data, then you probably cannot use that same data to feed a digital twin, drive predictive quality alerts, or support a control strategy change.

RTRT readiness, operationalized as a KPI, is best expressed as a percentage of critical quality attributes for which the site has:

• PAT sensor coverage sufficient to measure or model the attribute in-line
• A qualified process model that predicts the attribute with documented error bounds
• End-to-end traceability from sensor output to the release decision
• A change-control workflow for both the sensor calibration and the underlying model

Sites often discover that data silos, inconsistent calibration regimes, and fragmented software make it difficult to trace measurements from sensor output to release decision. Without end-to-end traceability and harmonized data management, real-time insights cannot be relied upon for compliance-critical functions.¹⁰ Measuring RTRT readiness surfaces those gaps well before a filing conversation forces them into the open.

What RTRT readiness reveals about the rest of the plant

Several critical concepts at the core of continuous manufacturing (predictive risk assessment, real-time release testing, and dynamically managed residence time distribution) all depend on deep, rich, and thoroughly contextualized process data. That contextualization is exactly the thing most legacy manufacturing environments do not have. Sensors exist, but the data flows into historians without process context. Models exist, but not in a registry that quality can inspect. Traceability exists, but only if a human recreates it manually for each investigation.

A site that scores well on RTRT readiness has, by definition, closed most of those gaps. That is why we recommend measuring it even in facilities with no near-term intent to file an RTRT-based product. The score is a proxy for something more fundamental: the maturity of the data pipeline that every advanced use case (predictive maintenance, closed-loop control, digital twin scheduling, batch genealogy for pharmacovigilance signal detection) will also need to draw on. Deferring the measurement until a filing conversation forces it is the classic pattern that leads to a scramble at the wrong time, with the wrong people in the room.

KPI 5 & 6: Digital Twin Fidelity and MES vs LIMS Reconciliation

KPI 5: Digital twin fidelity

A digital twin is a dynamic virtual model of a piece of equipment, a process, or an entire facility that continuously updates using real-time operational data. In pharma, digital twins are increasingly used for scheduling, yield prediction, tech transfer, and predictive quality. Recent industry analysis reports that digital twins can accelerate engineering and validation timelines by 40 to 70 percent when applied well.¹¹

The catch is that a digital twin is only as good as the data it is trained on and the discipline around updating it. Model fidelity must be reviewed periodically under change control to ensure data integrity and audit readiness as processes evolve.¹² Twins trained on poor quality data will generate poor quality predictions. And “poor quality” here does not just mean noisy or missing. It means unaligned with the process as it actually runs today.

Digital twin fidelity as a KPI combines three components:

1. Prediction accuracy: the rolling mean absolute error of the twin’s predictions against actual batch outcomes, tracked by critical quality attribute
2. Model currency: the age of the last qualified model refresh, versus the site’s own governance threshold
3. Change control coverage: the percentage of process changes (equipment, material, procedure) in the last review period that triggered a documented review of the affected twin models

KPI 6: MES vs LIMS reconciliation

MES-LIMS reconciliation may be the single most under-appreciated data quality indicator in pharma manufacturing. MES and LIMS, together with an eQMS, are widely described as the “triple pillars” of data integrity in a GMP-regulated environment.¹³ When they disagree, the disagreement is almost always resolved by a human, often under time pressure, often without a proper audit trail.

Reconciliation, as a KPI, measures how well the two systems agree on the facts of a batch without human intervention. Practically, it is a percentage of batches in the reporting period where:

• Sample identifiers reconcile automatically between MES and LIMS with no manual bridging
• Quantities, timestamps, and analyst signatures match to within the site’s tolerance
• Discrepancies, when they occur, are captured in a controlled reconciliation record with root cause and disposition

MES-LIMS integration is described in industry analysis as a “game-changer” for data integrity, because it provides real-time data synchronization that eliminates the risks caused by manual recording and fragmented documentation.¹⁴ Bringing digital twins into the picture creates a closed feedback loop: process data informs simulation models, simulation outputs guide process adjustments, and adjusted parameters are executed and documented through MES with full audit trail integrity.¹³

The hidden cost of manual reconciliation

Sites that have not measured reconciliation performance rarely appreciate how much manual bridging work happens between MES and LIMS every day. It shows up as production support tickets, as ad hoc spreadsheets maintained by senior analysts, as batch record notes that say “see LIMS attachment” without a controlled link. Each of those is a data integrity risk that will never be surfaced by an ALCOA+ audit, because the records themselves are compliant. What is not compliant is the process that ties them together, and that process is invisible until reconciliation is instrumented as a KPI.

Once instrumented, most sites find that manual reconciliation consumes far more analyst time than they expected, and that a meaningful fraction of that time is spent resolving errors that could have been prevented by better integration at the source. Reconciliation performance therefore becomes a lever for both quality and productivity, and it gives the CFO a reason to fund the integration work that quality has been asking for for years.

A Weighted Manufacturing Data Quality Scorecard Template

The six KPIs are not equally important, and pretending they are creates a scorecard that leadership stops trusting. The weightings below are a starting point, calibrated for a typical mid-cap pharma site with a mix of legacy batch and modern continuous operations. Sites with a heavier RTRT or continuous manufacturing footprint should tilt weight toward KPIs 4 and 5. Sites still stabilizing basic MES-LIMS integration should tilt weight toward KPI 6.

Scoring approach

Each KPI is scored on a 0 to 100 scale against its target, with a defined ramp. A KPI hitting target scores 100. A KPI at half of target scores 50. A KPI in known critical territory (for example, MES-LIMS auto-reconciliation below 70%) scores 0 and forces an escalation in that month’s management review regardless of the composite score.

The composite score is a weighted average of the six KPI scores. It is expressed on the same 0 to 100 scale, with the following bands:

Combining leading and lagging indicators

The scorecard deliberately combines leading and lagging indicators, following the balanced-scorecard tradition where predictive measures such as process capability and RTRT readiness are combined with outcome-based measures such as OOS trend accuracy and reconciliation performance.¹⁵ KPIs 1, 4, and 5 are leading indicators of future manufacturing performance. KPIs 2, 3, and 6 are lagging indicators of how well the current data pipeline is holding together. Both groups matter, and the weighting keeps either group from dominating.

Rolling the Scorecard Up to a Board-Level Indicator

Boards do not want a six-KPI dashboard. They want one number, a trend, and a clear signal of when to be worried. But the number they see should be defensible enough to hold up to a question from a director with a manufacturing background.

Our recommended approach is a single Manufacturing Data Trust Index (MDTI), calculated as follows:

The board pack shows:

• The current MDTI, on a 0 to 100 scale, with a 12-month trend line
• The two KPIs contributing most to any decline
• Any KPI in critical territory, with the associated remediation plan and owner
• A single narrative sentence from the head of quality or the CIO summarizing “what changed and what we are doing about it”

This approach echoes the FDA’s own direction of travel. Approximately 30% of FDA warning letters cite inadequate monitoring of KPIs as a major compliance issue, and the agency’s quality metrics reporting program has explicitly invited companies to submit certain quality KPIs, such as lot acceptance rate and right-first-time rate, to proactively identify issues before they affect patients.¹⁶ A defensible board-level indicator anchored in a real scorecard is exactly what regulators, and increasingly boards, are looking for.

A 12-Month Implementation Roadmap

Most mid-cap manufacturers can stand up a defensible version of this scorecard inside a year, if they resist the temptation to make it perfect on day one. The roadmap below has worked for the sites we have supported through this kind of build.

Where roadmaps most often fail

Two failure modes dominate. First, teams try to make the scorecard perfect before publishing it internally, which starves the initiative of the pressure it needs to succeed. Second, teams push the RTRT readiness inventory to a later phase, on the theory that it is a “digital” initiative rather than a data quality initiative. That framing is wrong. RTRT readiness is one of the highest-signal indicators of manufacturing data maturity you can measure. Delaying it delays every other advanced use case.

A third, less obvious failure mode is treating the scorecard as owned by quality alone. In practice, three of the six KPIs (RTRT readiness, digital twin fidelity, and MES-LIMS reconciliation) depend as much on IT, engineering, and data leadership as they do on quality. A scorecard that lives only in the quality organization tends to score poorly on exactly those three, because the people who could move the score are not accountable for it. The best implementations we have supported set the scorecard up as a joint quality-IT-engineering artifact, with a single owner (often a director of manufacturing systems or a fractional data leader) who is accountable for the composite score but not for any individual KPI.

Conclusion

Regulatory submissions are the visible part of pharma manufacturing data quality. Everything below the waterline (process capability from raw batch data, deviation and OOS trend integrity, RTRT readiness, digital twin fidelity, and MES-LIMS reconciliation) is where operational risk actually lives. A scorecard built around those six KPIs, with a defensible weighting scheme and a floor-adjusted composite, gives leadership a way to see the risk without drowning in dashboards. Rolled up to a single Manufacturing Data Trust Index with a 12-month trend and a narrative sentence, it also gives the board something they can actually use.

Sakara Digital works with pharma and biotech organizations building this kind of manufacturing data quality program, particularly in mid-cap environments where MES, LIMS, PAT, and emerging digital twin work all sit in different governance silos. If you are exploring what a Manufacturing Data Trust Index would look like for your site, or want an independent perspective on where to start on the six-KPI scorecard, we are happy to have that conversation.