Why Your Executive KPI Has Three Different Numbers: Three Reporting Quick Wins

Three Reports, Three Numbers, One Meeting

The pattern is familiar enough that most quality and IT leaders recognize it from the first sentence. A KPI appears on the site operations dashboard. It appears again in the monthly quality report that goes to the leadership team. It appears a third time in a slide prepared for the board or for a corporate quality review. The three values do not match. Somebody notices, usually in the meeting rather than before it, and the next twenty minutes are spent trying to work out which figure is right.

It happens to every KPI that senior leaders actually care about: right-first-time, deviation closure rate, batch cycle time, complaint rate, on-time release. These are the metrics with the widest audience, which is exactly why they get rebuilt independently in several places. A metric that only one team uses tends to have one calculation. A metric that five teams use tends to have five.

There are only three underlying causes, and it helps to name them separately because each has a different fix.

Cause one: different systems

The site dashboard pulls deviation records from the quality management system. The corporate report pulls from a data warehouse that was loaded from the same quality management system, but through an extract written four years ago that filtered out a record type nobody remembers. The board slide was built in a spreadsheet by someone who exported a report and added the sites that were missing. All three numbers describe reality. They describe different extracts of it.

System divergence is the easiest cause to detect and often the easiest to fix, because the difference between two extracts can usually be reconciled record by record. It is also the cause people jump to first, which is a problem, because it is not the most common one.

Cause two: different definitions

This is the one that does the real damage. Right-first-time can mean batches released with no deviation. It can mean batches released with no critical or major deviation. It can mean batches that cleared quality review without a query being raised. It can be counted on batches manufactured in the period, batches released in the period, or batches that reached final disposition in the period. Each of those is a defensible definition. Each produces a different number from identical source records.

Definition divergence is hard to see because nothing looks broken. Two reports run correctly, against the same system, and disagree. Unless someone opens the calculation logic in both, the disagreement looks like a data quality problem when it is a semantics problem.

Cause three: different cut times

The site dashboard refreshes nightly. The monthly report is cut on the fifth working day. The board pack is assembled a week before the meeting from whatever was current at the time. In a quality system where investigations close continuously and records are back-dated to the event, a metric cut on the fifth working day and the same metric cut on the fifteenth will differ, and the later cut is usually the worse-looking one because late-closing investigations are late for a reason.

What all three causes share is that none of them is a data integrity failure. The underlying records are fine. The failure is in the layer between the record and the slide, and that layer has almost never been documented, reviewed, or subjected to change control the way the source systems have. A validated electronic batch record system carries full traceability. The spreadsheet that turns its output into a percentage on a board slide usually carries none.

What it does to leadership confidence

The practical damage is not the twenty minutes lost in the meeting. It is what happens afterward. Once a leadership team has been burned twice by conflicting numbers, three things follow. Executives start asking for the raw data rather than the summary, which pushes analytical effort back onto teams that were supposed to be relieved of it. Decisions get deferred pending reconciliation, which turns a monthly review cycle into a quarterly one. And the quality organization loses standing in exactly the forum where it most needs to be believed.

There is also a regulatory dimension that leaders tend to underestimate. Management review of the pharmaceutical quality system is an expectation under ICH Q10, and the annual product review is a requirement under Part 211. If an inspector asks how a performance indicator presented to senior management was derived, and three versions of that indicator exist with no documented reconciliation, the answer is uncomfortable. The issue is not that a number was wrong. It is that the company cannot demonstrate which number management actually reviewed.

What Eleven Years of FDA Quality Metrics Work Actually Proved

Before proposing fixes, it is worth understanding how hard this problem is, because the evidence is public and most people in industry have never read it.

On 28 July 2015, FDA published a draft guidance for industry titled Request for Quality Metrics, under docket FDA-2015-D-2537, together with a Federal Register notice and a public meeting.¹ The draft proposed a mandatory program in which manufacturers would submit data at product level, from which FDA would calculate four primary metrics. The intent was stated plainly: use the data for risk-based inspection scheduling, to help predict and mitigate drug shortages, and to encourage modern quality management systems.²

The comments were extensive. In November 2016, FDA published a revised draft, Submission of Quality Metrics Data, which explicitly states in its own footnotes that in response to comments received in the public docket, FDA was replacing the 2015 draft with the revised version.³ The revised draft did two things: it dropped the mandatory framing in favor of a voluntary reporting phase, and it reduced the calculated metrics from four to three.⁴

The three surviving metrics, in FDA’s own words in that draft, were defined as follows:

Read those definitions closely and the reason this is difficult becomes visible. Every one of them contains at least one term that a company has to interpret before it can produce a number. What counts as a lot started? What counts as an accepted lot? Is a lot released with an unexpectedly low yield accepted or not? FDA had to write a glossary to answer some of these, defining a lot as a batch or a specific identified portion of a batch having uniform character and quality within specified limits, and defining an accepted lot as a started lot that has been released for distribution or for the next stage of processing.³ The agency also published a separate technical conformance guide in 2016 to specify the data format.⁵

FDA’s own selection criteria for the data set are instructive. The agency said it wanted objective data to provide consistency in reporting, of the type contained in records subject to inspection, and valuable in assessing the overall effectiveness of a pharmaceutical quality system without an undue reporting burden.³ Consistency in reporting was the first criterion, and it is the one that proved hardest.

The industry pilot that tested the definitions

While FDA was drafting, ISPE ran a two-wave pilot program, working with McKinsey and Company, to test whether a standardized set of quality metrics could actually be collected and reported. Wave 1 was reported in June 2015 and covered 44 sites from 18 companies. Wave 2 started in July 2015 and reported in June 2016, with 21 participating companies; across both waves the pilot covered 83 sites from 28 companies.⁶

Wave 2 was deliberately redesigned once FDA’s 2015 draft appeared, so that it could test the proposed FDA metrics as defined and, at the same time, test alternative definitions applied to the same data. Three of the four FDA metrics were evaluated. The fourth, an annual product review or product quality review on-time rate, was dropped because Wave 1 had found it did not differentiate between sites.⁶

The findings deserve to be quoted in substance because they are the strongest available evidence that definition, not data, is the constraint:

One specific finding is worth carrying into your own organization. The pilot compared two candidate denominators for lot acceptance rate: lots attempted and lots dispositioned. It found the two counts highly related but different in magnitude, and the relationship differed by site type. Across 18 solid dose sites the fitted slope between the two counts was 1.57 with an R-squared of 95 percent. Across 18 sterile sites the fitted slope was 1.03 with an R-squared of 59 percent.⁶

Read that plainly. Choosing one denominator rather than the other changed the size of the number materially, and it changed it by a different amount depending on the type of site. Any cross-site comparison built on top of an unspecified denominator was comparing site types, not site performance.

Where the initiative stands today

In June 2018, FDA announced two voluntary pilot programs in the Federal Register: a quality metrics site visit program for CDER and CBER staff, and a quality metrics feedback program for establishments with existing metrics programs.⁷⁸ In March 2022, FDA established a public docket describing considerations for refining what it then called the Quality Metrics Reporting Program, drawing on lessons from the 2018 pilots and on stakeholder feedback about the 2016 revised draft. Comments closed on 7 June 2022.⁹

As of this writing, the position is unchanged in one important respect. The 2016 revised draft guidance still carries the standard header stating that it is a draft, not for implementation, containing nonbinding recommendations.¹⁰ No quality metrics guidance has been finalized. No mandatory reporting requirement is in force. FDA’s public page on quality metrics for drug manufacturing still describes the program in these terms.⁴

The useful lesson from this history is not about regulatory timing. It is about difficulty. A regulator with statutory authority, a technical conformance guide, an industry association, a consulting firm, and 28 volunteering companies spent years on this and could not converge on three definitions. Your organization has almost certainly spent less effort on the definition of right-first-time than that. The three numbers in your deck are not a sign of incompetence. They are the normal outcome of an unwritten definition.

What ICH Q10 and Part 211 Ask of Management Review

The regulatory framing that does apply gets less attention than a reporting mandate would, but it is more directly relevant to the meeting where the three numbers appear.

ICH Q10 describes a pharmaceutical quality system model across the product lifecycle and sets an expectation that management reviews the performance of the system. The model calls for monitoring of performance indicators, periodic management review of the quality system, and documented outcomes from that review, including escalation of appropriate issues to senior management.¹¹ Q10 does not tell you which indicators to use or how to calculate them. That is deliberate. It leaves the choice to the company, which means the company owns the burden of showing that its chosen indicators are defined, consistent, and derived from records it can produce.

In the United States, 21 CFR 211.180(e) requires that written records be maintained so that data in them can be used for evaluating, at least annually, the quality standards of each drug product, to determine the need for changes in specifications or in manufacturing or control procedures. The regulation names what that evaluation must include: a review of a representative number of batches, whether approved or rejected, and a review of complaints, recalls, returned or salvaged product, and investigations.¹² The annual product review, and its European counterpart the product quality review, is therefore a second place where the same underlying counts get aggregated, often by a different team, on a different calendar, with a different definition.

This is why organizations routinely end up with more copies of a KPI than they expect. The metric exists in operational reporting because sites need it weekly. It exists in management review because Q10 expects performance indicators. It exists in the annual product review because Part 211 requires an annual evaluation. It exists in the board pack because leadership asked for a trend. Four legitimate demands, four independent builds, one KPI.

The PDA has published useful practitioner work on this point, arguing that standardizing key performance indicators and metrics is a prerequisite for assessing the state of a pharmaceutical quality system, while acknowledging that good definitions are usually specific to an organization, particularly in how a KPI is calculated from raw data.¹³ Both halves of that statement matter. Standardization is necessary. Universal standardization across the industry has repeatedly failed. What works is internal standardization, written down, with a named owner.

That is the whole basis for the three quick wins below. None of them tries to solve metrics for the enterprise. Each of them takes one metric, or one report family, or one comparison, and makes it defensible inside a few weeks.

Quick Win One: Consolidate the Reports Carrying One KPI

Scope: two to three weeks. Team: one analyst, one quality subject matter expert, one report owner from each function that publishes the KPI. Before-and-after measure: the number of live reports, dashboards, and recurring decks that publish the chosen KPI, and the number of distinct calculation paths behind them.

Start with the inventory, because the count itself is usually the finding. Pick one KPI that senior leaders reference by name. Then find every place it is published.

Two practical notes carry most of the difficulty. First, renaming matters as much as retiring. If a site keeps a local right-first-time calculation because it needs a weekly operational signal, that is fine, but it should be called something else, because the moment two things share a name they will be compared. Second, resist the urge to build a new report. The consolidation quick win is about subtraction. A new dashboard added to an inventory of eleven produces twelve.

Quick Win Two: Run a Single-Number Audit on One Executive KPI

Scope: three to four weeks. Team: one analyst, one data engineer with access to the source systems, one quality subject matter expert. Before-and-after measure: the percentage difference between the highest and lowest published value of the KPI for the same period, before the audit and after the reconciliation.

The consolidation quick win tells you how many versions exist. The single-number audit tells you why they differ, by tracing one figure from the executive slide back to the source records and documenting every transformation on the way.

The lineage trace, worked

Take one KPI, one period, one reported value. Then walk backward. At each step, record what the value is at that point, what operation was applied, and where the operation is defined. Here is what a completed trace looks like for a deviation closure rate reported to leadership as 91 percent for a quarter.

A trace like this normally takes an analyst and an engineer three to five working days per KPI, and it usually turns up two or three findings that nobody knew about. In the example above, step 3 is the finding: an exclusion added over a year ago with no record of who approved it, which removes the slowest population of deviations from the denominator and therefore improves every downstream number. Step 4 is the second finding, and a subtler one: a nightly incremental load that never re-reads amended records means late corrections never reach the report at all.

Running the audit

The sequence that works is deliberately narrow. Pick the KPI leadership questions most often, not the one that is easiest to trace. Pick a closed period so the underlying records are stable. Then work backward one layer at a time, and at each layer, reproduce the value independently before moving on. If you cannot reproduce the value at a layer, you have found the divergence and you stop and investigate rather than continuing the trace.

Two rules keep this from expanding. First, one KPI, one period. The temptation to trace three at once is strong and it is what turns a four-week pilot into a six-month project. Second, document as you go, in the table format above, rather than writing the report at the end. The table is the deliverable.

The definition template

The audit produces a reconciled number. What makes the reconciliation stick is writing the definition down in a form that is specific enough to be implemented identically by two people who have never met. The following template is short on purpose. Anything longer does not get filled in.

Three fields on that template do more work than the rest. The date field, because period assignment is the least visible source of divergence and almost nobody records it. The cut rule, because a metric that gets restated every month without announcement can never be reconciled against an older report. And change control, because the whole point of the exercise is to prevent the definition drifting again the moment someone reasonable makes a reasonable change.

Quick Win Three: Harmonize Site Comparison Before Anyone Compares Sites

Scope: three to four weeks. Team: the quality lead from each site, one analyst, one corporate quality owner to arbitrate. Before-and-after measure: the number of sites using an identical definition, calculation, and reporting calendar for the chosen KPI, before and after.

This is the quick win with the highest political content and the highest payoff. The moment a network of manufacturing sites is ranked on a single KPI, the ranking drives attention, investment, and sometimes performance conversations about named individuals. If the underlying definitions differ, the ranking is measuring definitional choices rather than performance, and the sites usually know it before corporate does.

The ISPE pilot finding described earlier is the evidence to bring into this conversation. Two defensible denominators for the same lot acceptance rate produced counts that differed in magnitude, and differed by different amounts at solid dose sites than at sterile sites.⁶ A network with both site types, comparing lot acceptance rate without a common denominator, is comparing dosage form.

What divergence looks like in practice

A typical starting position across four sites, for right-first-time, looks like this. None of these sites is doing anything improper. Each definition was set locally for a local reason.

Every cell that differs is a reason two sites cannot be compared. Site C’s exclusion of rework and its use of a fiscal calendar make its figure structurally higher and structurally offset in time. Site D’s tenth working day cut captures late closures that Site A’s third working day cut does not, which makes Site D look worse for a reason that has nothing to do with how Site D operates.

The harmonization sequence

Step 4 is the one most often skipped and the one that most often causes the harmonization to fail. If the first comparable report is also the first report under the new definition, every site can plausibly claim its number moved because of the change, and the exercise produces a new argument rather than ending an old one.

Presenting the Reconciled Number Without Undermining Trust

This section addresses the part of the work that has nothing to do with data and determines whether the pilot succeeds. You have reconciled a KPI. The reconciled value differs from what leadership has been shown for the past several quarters, and it is usually worse, because undocumented exclusions and early cuts tend to flatter. Now you have to present it.

The failure mode is to present the reconciliation as a discovery of error. That framing invites two damaging questions: what else has been wrong, and who is responsible. Neither question has a good answer, and both distract from the finding that actually matters, which is that the organization now has a number it can defend.

What to say

Lead with the decision, not the discrepancy. Open by stating the reconciled figure, the definition behind it, and what it means for the business. Then explain the reconciliation as the reason the figure can now be relied on, rather than as a correction of past reporting.

Be specific about why previous figures differed, in neutral terms. An exclusion was applied at the warehouse layer and was not carried into the definition. Two sites used different closure targets. The figure was cut on the fifth working day and late closures were never re-read. These are mechanical explanations, and mechanical explanations are easier to accept than judgments about people.

Say clearly what has not changed. The underlying operational performance did not move. The records were always correct. What changed is the calculation applied to them and, critically, the fact that the calculation is now written down and under change control.

What not to say

Do not say the old numbers were wrong. In almost every case they were not wrong, they were differently defined, and the distinction is both accurate and considerably easier for an audience to absorb. Do not present a long list of every discrepancy found during the trace; present the two or three that account for most of the difference and keep the rest in the appendix. Do not commit, in that meeting, to reconciling every other KPI. The value of a quick win is that it is bounded, and a leadership team energized by one good result will happily extend the scope until it stops being a quick win.

Finally, do not let the reconciled number be presented by an analyst alone. The person who owns the definition should present it, because the durable message is that this metric now has an owner. That is the change that prevents the three numbers coming back.

Knowing the Pilot Worked, and What Comes Next

Every quick win in this series carries a before-and-after measure that can be stated in one line and verified by someone who was not involved. For these three, the measures are as follows.

Those measures are deliberately unglamorous. That is the point. A leadership team that has been promised data transformation before is more persuaded by a stated count that moved from eleven to two than by a maturity assessment.

What to do next, and what not to

The natural next step after one successful reconciliation is the second KPI, then the third. Do them one at a time, and use the same definition template, because a set of five completed templates is the beginning of a real definition library and five differently formatted documents is not.

The step to avoid is converting the quick wins into a program. The moment this becomes a data governance initiative with a charter, a steering committee, and a two-year roadmap, it acquires overhead that exceeds the value of the next few reconciliations, and it starts competing for attention with everything else on the portfolio. The organizations that get furthest with this work keep it small deliberately, run two or three reconciliations a year, and let the definition library accumulate.

Where these pilots point

There is a longer-term payoff that is worth stating even though it should not be the justification for the pilot. Every one of these quick wins produces exactly the artifacts that an AI or advanced analytics initiative later requires: documented metric definitions, traced lineage from source record to reported figure, named owners, and change control on the calculation layer. Organizations that attempt to apply machine learning to quality data before doing this work generally discover that the model has learned the definitional inconsistency rather than the process.

But the immediate justification is simpler and stronger. Leadership needs to be able to look at a number and act on it. Three numbers means no decision. One number, defined, owned, and traceable, means the management review can be about the business again.

Conclusion

The three-numbers problem is not evidence of a broken quality system or a failed data strategy. It is evidence that the layer between the record and the slide was never treated with the same discipline as the record itself. The regulatory history makes the point better than any argument could: FDA has worked on pharmaceutical quality metrics since 2015, published a draft, replaced it, run pilot programs, opened a docket, and still has no final guidance, largely because agreeing what a metric means turned out to be harder than collecting the data behind it. When 28 companies and a regulator cannot converge on the definition of a lot acceptance rate, an unwritten definition inside a single company is going to produce more than one number every time.

The fix does not require a program. It requires picking one KPI that leadership actually uses, counting how many places publish it, tracing one published figure back to source records, writing the definition down with an owner and a change control route, and then presenting the reconciled figure as the beginning of something reliable rather than the correction of something wrong. Two to four weeks per pilot, a measure that moves, and an artifact you can hand to an inspector. Do that three times and the reporting layer starts to earn the confidence the source systems already have.

Sakara Digital works with pharma and biotech organizations building exactly this kind of defensible reporting layer, from metric definition and lineage tracing through to the governance that keeps definitions from drifting again. If you are looking at a KPI with more than one value and want an independent perspective on where to start, we are happy to have that conversation.

For Further Reading