Why Most Quality Culture Metrics Measure the Wrong Layer

Ask a site head how the quality culture is doing and the answer usually comes back as a dashboard. Deviations per thousand batches. Percentage of investigations closed on time. CAPAs overdue. Number of observations at the last inspection. These are reasonable things to track, and some of them belong in any quality report. The trouble is what happens when they are used as the measure of culture and the people being measured are the leaders who are supposed to build it.

Every one of those numbers is an outcome. It is the end result of hundreds of small decisions made by operators, supervisors, analysts, investigators, and managers over a period of months. A leader cannot reach into that chain and change the outcome directly. What a leader can change is a much shorter list: what gets discussed and decided in management review, where money and headcount go, how fast they respond when someone raises a problem, what they say and do when the news is bad, and what they reward. Those inputs shape the behavior of everyone else, and that behavior eventually shows up in the outcomes.

When the scorecard measures only outcomes, two things go wrong. First, leaders get no feedback on the part of the system they control, so they cannot tell whether their own actions are working until a year has passed. Second, the outcome numbers become targets, and targets on reported quality events create pressure to report fewer of them. A site can improve its deviation count without improving anything else.

What the Regulators Say Leadership Owns

The regulatory framework is clear that culture is a management responsibility. ICH Q10 lists what management should do in a pharmaceutical quality system. Among other duties, management should “Ensure a timely and effective communication and escalation process exists to raise quality issues to the appropriate levels of management,” conduct management reviews, advocate continual improvement, and commit appropriate resources.1 Q10 also says performance indicators that measure progress against quality objectives should be established, monitored, communicated, and acted on.1

The PIC/S guidance on data management and integrity (PI 041-1) is more direct about behavior. In its section on quality culture, it lists ways management can foster that culture, including leading by example, staying actively involved in operations, setting realistic expectations that account for the pressures on employees, allocating enough resources, and using fair and just consequences and rewards.2 Each item on that list is something a leader does, not something a leader reports.

FDA’s QMM program frames it the same way. The program’s 2023 white paper describes five practice areas, and the first is management commitment to quality. It says management bears the responsibility for setting the tone and modeling a culture of quality, and that an assessment may look at the effectiveness of management review, “including how often they perform their review, to what depth they review data, and how they use the outputs of the review process to initiate continual improvements.”4 That sentence is close to a specification for a leadership behavior metric.

Three Layers, Sorted by Control

A useful way to organize quality culture metrics is by how directly leadership controls each one. That gives three layers.

LayerWhat It MeasuresLeadership ControlHow Fast It MovesExamples
1. Leadership behaviorWhat leaders decide, fund, and sayDirectWeeksManagement review decisions, time to resource an escalation, leader response to bad news
2. Speak-up signalsWhether people raise problems early and honestlyIndirect, through layer oneOne to three quartersReporting rate by severity, event-to-record lag, self-identified share of findings
3. OutcomesWhether the quality system learnsDistantTwo to four quarters or longerRepeat deviations, CAPA effectiveness, recurrence after closure

The point of the layers is not to replace outcome metrics. It is to stop using them alone. Layer one tells leaders whether they did what they said they would do. Layer two tells them whether the organization believed it. Layer three tells them whether it made a difference. A site that improves layer three while layer two falls should be suspicious of its own good news.

What the Research Says Culture Predicts

Before building a scorecard, it is worth knowing what the evidence supports and what it does not. There is a reasonable body of pharma-specific research linking quality culture to quality system performance, but most of it is correlational, much of it is self-reported, and none of it hands you a ready-made metric set.

PDA’s Surrogate Attributes

The Parenteral Drug Association ran an anonymous global survey of quality culture in the industry and published the results in 2015 as the Quality Culture Survey Report. The authors were testing whether certain quality behaviors relate to certain quality attributes, and whether those attributes could serve as stand-ins for measuring culture. According to the published abstract, the analysis suggested that certain attributes drive good behaviors and that the relationship holds regardless of where a site is located.6

The five candidate surrogate attributes the report named are worth reading as a list, because most of them are leadership actions: management communicating that quality is everyone’s responsibility, formal quality improvement objectives and targets, clear performance criteria for feedback and coaching, quality topics included in at least half of all-hands meetings, and collecting error prevention metrics.6 The authors linked these attributes to the management responsibility and continual improvement sections of ICH Q10.

The same report found that executives rated the current state of quality culture more favorably than other respondents did, with the largest gap on cross-functional vision.6 That finding matters for measurement design. If leaders and staff see the same culture differently, the gap between their answers is itself a metric, and a survey that only reports the site average hides it.

St. Gallen’s Link to Quality System Effectiveness

Researchers at the University of St. Gallen, whose operational excellence benchmarking approach FDA’s QMM white paper cites as one example of a program that evaluates quality culture,4 tested the link between culture and pharmaceutical quality system (PQS) performance using their benchmarking database. In a summary article, Friedli, Köhler, and Buess reported that “a high degree of PQS Effectiveness is accompanied with a high level of Cultural Excellence.”7 They described cultural excellence as a combination of quality behavior, quality maturity, and engagement measures, and reported that high-performing sites scored significantly higher on each. The database they drew on covered 339 manufacturing sites from more than 124 companies.7

The article also reported that quality maturity (the implemented elements of the quality system, such as methods and tools) explained 66 percent of the variation in quality behavior in their analysis, compared with 34 percent in the earlier PDA study.7 Read plainly, the systems a site puts in place and the way its people behave move together. That does not tell you which causes which, but it does suggest that behavior and system maturity should be measured side by side rather than in separate programs.

What the FDA-Funded Benchmarking Found, and What It Did Not

A 2022 paper in the AAPS Journal by Fellows, Friedli, and colleagues reported on an FDA-funded quality benchmarking study run by Dun & Bradstreet with the University of St. Gallen. The study collected performance data and self-assessed quality practices from 208 manufacturing establishments.8 Its headline finding was a significant positive correlation between delivery performance and the application of QMM principles tied to technical production.

One of the five practices that correlated with better delivery performance was CAPA effectiveness. Establishments reporting high implementation of CAPA effectiveness had a mean delivery percentile score of 49.4, against 42.2 for low implementers. Establishments with high implementation of four or more of the five practices, 28.8 percent of respondents, averaged 54.7 against 42.0 for the rest.8 The authors are careful to say that correlation does not imply causation, and that practices without significant correlations were not shown to be unimportant.

The honest reading for culture measurement is this: the strongest statistical signal in that study came from technical and tool-based practices, not from culture questions. That is not evidence that culture does not matter. It is evidence that self-assessed culture is hard to measure well, which is the problem this article is trying to address.

208Establishments in the FDA-funded quality benchmarking study reported in the AAPS Journal8
9Establishments CDER assessed in the first year of its QMM prototype program, 20245
84Approximate open and past-due deviation investigations FDA documented at one API site in a January 2025 warning letter14

The Safety Culture Evidence on Reporting

The most useful single finding for anyone designing quality culture metrics comes from outside pharma. In a 1996 study of eight hospital nursing units, Amy Edmondson found that units with stronger nurse manager leadership and better working relationships reported higher rates of medication errors, not lower.10 Her explanation was that detected error rates depend on two things: how many errors happen and how willing people are to report them.

Ron Westrum’s typology of organizational cultures makes a related point. He classified organizations by how they handle information, and argued that information flow can predict how an organization behaves when signs of trouble appear. In his 2004 paper, he reported that this information culture is associated with error reporting and with performance, including safety, while noting that the connection needed more study before it could be considered definitive.12

Put those two findings next to a pharma deviation dashboard and the problem is obvious. A falling deviation count can mean fewer problems. It can also mean fewer reports. Without a second measure, there is no way to tell which.

The Limits of This Evidence

None of this research gives a validated, ready-to-use set of quality culture metrics. The PDA work is survey-based. The FDA-funded benchmarking relied on self-assessed practices. The strongest reporting evidence comes from hospitals, not manufacturing sites. What the evidence does support is a design principle: measure what leaders do, measure whether people speak up, and only then read the outcome numbers. That is the structure of the rest of this article.

Layer One: Leadership Behaviors You Can Count

Leadership behavior sounds hard to measure. In practice, much of it leaves a written record. Management review has minutes and action logs. Resource requests have dates and decisions. Escalations have timestamps. Floor visits can be logged with what was found and what happened next. The measures in this layer come from records the site already keeps or could keep with very little effort.

Management Review as a Behavior Record

ICH Q10 sets out what management review of process performance and product quality should include: results of inspections and audits, periodic quality reviews (complaints, recalls, process monitoring, and the effectiveness of changes, including those from CAPA), and any follow-up actions from previous management reviews. It also says the review should identify actions such as process improvements, provision or realignment of resources, and capture and sharing of knowledge.1

Most sites can show that management review happened. Fewer can show that it decided anything. The useful measures look at what came out of the meeting rather than whether it was held:

  • Decision rate. Of the items presented, how many ended with a named decision, owner, and date, rather than “noted” or “continue to monitor”?
  • Carry-over rate. How many actions from the previous review are still open, and how many have been carried forward more than twice?
  • Resource outcomes. How many review cycles produced a change in resources (people, budget, equipment, training)? Q10 names resource allocation as an expected outcome.1 A management review that never moves a resource is a reporting meeting.
  • Depth. Did the review look at the underlying data for at least one topic, such as the individual repeat deviations behind a trend, or only at summary charts? QMM explicitly considers the depth at which management reviews data.4

These measures are cheap to collect. Someone reads the minutes after each review and fills in four fields. Over a year, the pattern shows whether the leadership team uses the review to run the quality system or to receive a briefing on it.

Time From Escalation to Resource

Q10 expects a timely and effective escalation process.1 Most sites measure the first half of that, how fast an issue reaches management, and never measure the second half: how fast management responds with something concrete. A useful metric is the median number of days between a formal escalation (a quality risk raised to the site leadership team, a resource request from the quality unit, an alert from trending) and the first concrete response, such as approved headcount, a funded fix, a changed schedule, or a documented decision not to act with a reason.

The Sanofi warning letter of January 2025 shows why this matters. FDA asked the firm to describe how top management supports quality assurance and reliable operations, “including but not limited to, timely provision of resources to proactively address emerging manufacturing/quality issues and to assure a continuing state of control.”14 FDA is asking about the speed of the leadership response, not only the existence of an escalation path. A site that tracks escalation-to-resource time has a direct answer to that question.

Floor Presence With Follow-Through

Many sites count gemba walks (structured leadership visits to where the work happens). The count alone says little. A leader who walks the floor weekly and never closes anything they see is teaching people that raising issues during a walk is pointless. The better measure pairs presence with closure: of the issues raised to leaders during floor visits, how many were resolved or answered within an agreed time, and did the person who raised the issue hear back?

ISPE’s APQ guide on cultural excellence treats “Gemba and Employee Engagement” as one of the aspects it evaluates, alongside leadership and vision, leading quality indicators, and proactive oversight, review, and reporting.9 The full guide is sold by ISPE, so this article does not attribute specific measures to it, but the pairing of floor presence with engagement fits the closure measure described here.

How Leaders Respond to Bad News

This is the hardest leadership behavior to measure and the one that matters most. People decide whether to report a problem based on what happened to the last person who did. PIC/S PI 041-1 describes “open” cultures as those “where hierarchy can be challenged by subordinates, and full reporting of a systemic or individual failure is a business expectation,” and “closed” cultures as those where reporting failure or challenging hierarchy is culturally more difficult.2 It notes that closed cultures may need more oversight and secondary review to reach the same level of control.

Some proxies that can be tracked:

  • Investigation conclusions naming individual blame. The share of investigations closing with retraining or discipline of an individual as the main corrective action. A high and stable share suggests the system looks for people to fix rather than processes.
  • Use of the confidential channel. PIC/S expects a confidential escalation program that lets personnel raise possible breaches with senior management without consequence, including a route for concerns about senior management itself.2 Track volume and, more important, whether reports led to visible action.
  • Leader recognition of reporting. How often leaders publicly credit someone for raising a problem early. PIC/S says behaviors that conform to good data management practice should be actively encouraged and recognized.2
Leadership Behavior MetricDefinitionData SourceLinked Expectation
Management review decision rateShare of review items closing with a decision, owner, and dateReview minutes and action logICH Q10 3.2.4 and 4.3; QMM management commitment
Action carry-overPrior-review actions still open; count carried more than twiceAction logICH Q10 3.2.4 (follow-up actions)
Escalation-to-resource timeMedian days from formal escalation to first concrete responseEscalation log, approvalsICH Q10 2.1 and 2.4; PIC/S 6.5.2 and 6.6
Floor issue closureShare of issues raised during leader visits closed or answered within the agreed timeVisit logPIC/S 6.3.3 (active involvement)
Individual-blame shareShare of investigations whose main corrective action is retraining or discipline of an individualDeviation and CAPA systemPIC/S 6.3.3 (fair and just consequences)

Layer Two: Signals That Show Whether People Speak Up

The second layer tests whether the organization believes what leadership is doing. These measures move more slowly than layer one, but faster than outcomes. They are also where the most common misreadings happen.

Deviation Reporting Rates, Read by Severity

A rise in deviation reports is often the first sign that a culture is getting healthier, because people who used to fix small problems on their own and move on start writing them down. That is the Edmondson finding applied to manufacturing.10 The trouble is that a rise in reports can also mean the process is getting worse. The rate alone cannot separate the two.

The way to separate them is to read volume against severity. When a reporting culture improves, the growth comes mostly from minor events and near misses: things that would previously have gone unrecorded. Major and critical events, which are harder to hide, should hold steady or fall. When a process deteriorates, the growth shows up across severities, including the serious ones. Reading the two together gives four patterns.

Likely healthy

Minor Reports Rising, Major and Critical Flat or Falling

People are recording small problems they used to absorb. The serious events, which are hard to hide, are not increasing. This is the pattern a culture program should produce.

Investigate the process

Reports Rising Across All Severities

Something in the process, materials, equipment, or staffing is changing. Treat it as a technical signal first, and check whether a recent change lines up with the rise.

Warning

Minor Reports Falling, Major and Critical Flat or Rising

The small events are disappearing from the record while the serious ones are not. The most common explanation is that people have stopped reporting what they think they can handle themselves.

Confirm before celebrating

Reports Falling Across All Severities

This may be real improvement. Confirm it with layer two lag and self-identification measures and with layer three repeat rates before reporting it upward as a win.

Two practical notes. First, normalize volume by activity (batches, lots, analytical runs, or shifts), or a busy quarter will look like a culture problem. Second, severity classification is itself something people can shift under pressure. If the share of events classified as minor rises sharply while the volume of major events falls, check whether classification practice changed before concluding anything about the culture.

Time to Escalate: Event Date Versus Record Date

Most deviation systems record when an event occurred and when it was entered. The gap between those two dates is one of the most useful culture signals a site has, and almost nobody reports it. When people trust the system, they report quickly. When they fear the response, they wait, try to fix it first, check with a colleague, or hope it goes away.

Track the median and the 90th percentile of event-to-record lag by area and by shift. A shift or department with a long tail is telling you something about its supervisor. Track the same measure for escalation from the record to the quality unit and from the quality unit to site leadership for events that meet escalation criteria. Q10’s expectation of a timely escalation process applies at each step.1 If a critical event takes days to reach the site head, the escalation procedure is not working, whatever the SOP says.

Who Finds the Problems

A mature quality system finds its own problems. One of the clearest culture signals is the share of significant issues the site identified itself, through its own deviation reporting, trending, self-inspection, or internal audit, compared with issues first identified by a regulator, a customer audit, or a complaint.

The Sanofi warning letter gives a direct example of the opposite. FDA cited a failure to ensure all production deviations are reported and evaluated, and wrote that “Multiple production deviations went unaddressed until our investigators informed facility personnel of the issues.”14 One example was a line used for aseptic processing touching the floor during setup, which the batch record specifically instructed personnel to prevent. Those events were visible to the people doing the work. They were not reported. The same letter documented approximately 84 open and past-due deviation investigations as of June 21, 2024, and stated that the findings indicated the quality unit was “not able to fully exercise its authority and/or responsibilities.”14

The self-identified share is easy to calculate once a site agrees on what counts as a significant issue. It also lines up with Q10, which lists self-assessment processes and external assessments as separate inputs to management review of the quality system.1 If the external column keeps finding things the internal column missed, the site has a speak-up problem, an internal audit problem, or both.

Near Misses and Good Catches

Near-miss reporting (recording events that could have caused a deviation but were caught in time) is the purest speak-up signal, because nobody is required to report a near miss. Volume is therefore almost entirely voluntary. A site with no near-miss reports does not have a perfect process. It has people who do not see the point of reporting.

The PDA survey’s fifth surrogate attribute, collecting error prevention metrics, fits here.6 The measures worth tracking are the volume of near-miss reports normalized by activity, the share that led to a recorded change, and the time from report to feedback to the reporter. The last one is the measure that keeps the others alive. People stop reporting near misses within a few months if they never hear what happened.

The reporting paradox in one sentence. When a culture program works, the first number to move is usually the one leadership is used to seeing go down, and it goes up. Leaders need to know this before the program starts, in writing, or the first good result will be read as a failure and the program will be redesigned to make the number fall again.

Layer Three: Outcomes That Confirm the Culture Changed

Outcome metrics belong in the scorecard. They are the evidence that leadership behavior and speak-up signals are producing a quality system that learns. They just should not be the only thing on the page, and they should not be the targets leaders are paid on.

Repeat Deviations

A repeat deviation is one with the same or a closely related root cause as a previous event that was supposedly corrected. The repeat rate is the most direct measure of whether investigations and CAPAs are fixing causes or only symptoms. It is also harder to game than a raw count, because hiding a repeat requires either not reporting it or deliberately classifying it as something new.

The definition needs care. Decide in advance what counts as related (same root cause category, same equipment, same process step, same product family) and how far back to look. Without a fixed definition, the repeat rate drifts as investigators learn which classification keeps the number down. The Sanofi letter also shows what FDA looks for here: it cited the firm for repeatedly deviating from its validated manufacturing process and found that investigations “failed to identify all potential contributing causes.”14

CAPA Effectiveness

CAPA effectiveness asks whether the corrective or preventive action stopped the problem from coming back. It is measured after the CAPA closes, through a planned check at a defined interval with defined success criteria. The FDA-funded benchmarking study found that establishments reporting strong CAPA effectiveness practice had better delivery performance, one of only five practices with a significant correlation.8

FDA also asks for it directly. In the Sanofi letter, the agency called one response inadequate partly because the firm did not “explain how the effectiveness of the CAPA will be determined,” and in another section asked for “the means of determining CAPA effectiveness.”14 A site that cannot show its effectiveness checks, with criteria set before the check was run, does not know whether its CAPA program works.

Two measures matter. The first is the share of closed CAPAs that received an effectiveness check on schedule. The second, more important, is the share of checks that found the action ineffective. A site where every effectiveness check passes is either very good or not checking hard enough, and the second explanation is more common.

Why Outcomes Cannot Be Steered Directly

Repeat rates and CAPA effectiveness move slowly. A change in leadership behavior this quarter might show up as fewer repeats a year from now, after the events have happened, been investigated, been corrected, and been checked. That delay is the reason outcome metrics are poor targets for leaders. By the time the outcome moves, the decision that moved it is long past, and the pressure to improve the outcome this quarter can only be applied to the reporting, the classification, or the closure date. That is how good outcome metrics turn into bad ones.

Staff Survey Items That Track Leadership Behavior

Surveys are the most common way sites measure quality culture, and the most common source of numbers nobody acts on. Sakara Digital has written separately about running a quarterly pulse survey with a short item set. This section focuses on a narrower question: how to write and read survey items so that they tell leaders something about their own behavior.

Start With Tested Items

Edmondson’s 1999 study of 51 work teams in a manufacturing company used a seven-item scale for team psychological safety. The items include “If you make a mistake on this team, it is often held against you,” “Members of this team are able to bring up problems and tough issues,” and “It is safe to take a risk on this team.”11 The same study measured team leader coaching with items such as whether the leader initiates meetings to discuss the team’s progress and is available for consultation on problems. Items like these have been used and studied for decades, which makes them a better place to begin than new items written from scratch.

The FDA QMM white paper points the same way. Under employee engagement and empowerment, it describes engaged employees as empowered “to identify, communicate, escalate, and initiate changes,” and says assessments may consider whether leadership creates an environment that encourages employees to share their thoughts and ideas.4 It also says more mature establishments foster a culture of active participation “without fear of reprisal.”4

Ask About Events, Not Attitudes

General attitude items (“Leadership is committed to quality”) produce high, stable scores that rarely change and rarely lead to action. Items that ask about a specific recent event are harder to answer politely and easier to act on. Some examples that tie directly to layer one behaviors:

  • The last time I raised a quality concern, I heard back about what happened to it.
  • In the past three months, I have seen a production or schedule decision changed because of a quality concern.
  • When I report a deviation, my supervisor’s first question is about what happened, not who did it.
  • I have seen a member of site leadership on my area’s floor in the past month, and something they saw was followed up.
  • I know what changed as a result of a deviation in my area in the past six months.

Each item maps to something in the leadership behavior table. If the floor closure metric says 90 percent of issues are closed on time and the survey says most people never hear back, the two measures disagree, and the disagreement is the finding.

Measure the Gap Between Leaders and Staff

The PDA survey found executives rated culture more favorably than other respondents.6 Ask leaders and staff the same items and report the gap by item. A large gap on “I heard back about what happened” means leaders believe they are closing the loop and staff do not experience it. That is a specific, fixable problem, which is more than a site average can tell you.

Small Sites and Anonymity

Many biotech manufacturing and laboratory teams are small. A survey cut by shift or area may have five respondents, which makes results both noisy and identifying. Set a minimum reporting group size in the site’s survey policy and report smaller groups only at the combined level. If people suspect their answers can be traced, the survey measures fear, not culture.

Metrics That Create Perverse Incentives

The social scientist Donald Campbell put the core problem plainly in a 1976 paper later reprinted in the Journal of MultiDisciplinary Evaluation: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”13 His example is uncomfortably close to pharma. Police departments judged on clearance rates (the share of crimes solved) faced pressure that, he wrote, led to simple evasions such as failing to record all citizens’ complaints or postponing recording them unless solved.13 Replace “complaints” with “deviations” and the mechanism is the same.

PIC/S says the same thing in regulatory language. Its guidance on management review of performance indicators states: “Caution should be taken when key performance indicators are selected so as not to inadvertently result in a culture in which data integrity is lower in priority.”2 It also lists “incentives for productivity in excess of process capability” among the indirect influences on employee behavior that management should understand and address.2

The test for any quality culture metric. Before adopting a metric, ask one question: what is the easiest way for a person to make this number look better without making the quality system any better? If the answer is “report less,” “classify down,” “close early,” or “check less hard,” the metric needs a counter-measure beside it, or it should not be a target.

Common Metrics That Get Gamed

Metric Used as a TargetHow It Gets GamedCounter-Measure to Pair With It
Deviation count reductionEvents go unreported, get fixed informally, or are logged as something elseMinor-to-major ratio; near-miss volume; self-identified share of findings
“Zero deviations” goalsStrong signal that reporting is unwelcome; reporting falls firstRetire the goal; replace with repeat rate
Investigations closed on timeRoot cause analysis cut short to meet the date; closure on a symptomRepeat rate; investigation depth review; CAPA effectiveness failure rate
CAPA closure rateActions designed to be easy to close rather than effectiveShare of effectiveness checks that found the action ineffective
Right-first-time bonusCorrections made off the record; errors not documentedEvent-to-record lag; audit trail review findings
Severity mix targetsEvents classified down to avoid major or critical labelsPeriodic independent reclassification sample
Survey score targetsManagers coach teams on answers; low scorers discouraged from respondingReport response rate and leader-staff gap; never tie scores to pay

Why Bonuses Make It Worse

Tying a leader’s variable pay to a quality outcome metric is the fastest way to turn the metric into a measure of reporting behavior. It is not that leaders are dishonest. It is that the pressure passes down the organization, and the people at the bottom of the chain are the ones who decide whether to write a deviation. They do not need to be told to report less. They only need to see that reports cause trouble for their boss.

If quality must be part of incentive design, tie it to layer one behaviors that the leader controls and that cannot be improved by reporting less: management review decision rate, escalation-to-resource time, completion of effectiveness checks with predefined criteria. Those numbers can be gamed too, but gaming them requires doing more of the work, not less.

Investigation Timeliness Deserves Its Own Warning

The single most common perverse incentive in pharma quality is the fixed investigation deadline used as a performance target. When on-time closure becomes the measure, investigators learn to close investigations on the first plausible cause. Sakara Digital has written in detail about why a single closure deadline is the wrong measure and what a risk-tiered alternative looks like. For culture measurement, the short version is that on-time closure should be reported alongside repeat rate, and a site whose on-time rate rises while its repeat rate also rises is getting faster closures and more recurrence at the same time.

Building and Running the Scorecard

A quality culture scorecard does not need many measures. It needs the right few from each layer, a fixed definition for each, and a place in management review where the leadership team looks at the layers together and decides something.

1

Agree on Definitions Before Collecting Data

Write down exactly how each measure is calculated: what counts as a repeat, what counts as self-identified, which date fields define event-to-record lag, what counts as a concrete response to an escalation. Fix the definitions under change control so they cannot drift.

2

Pick Two or Three Measures per Layer

For layer one, management review decision rate and escalation-to-resource time are good starting points. For layer two, severity-split reporting rate and event-to-record lag. For layer three, repeat rate and CAPA effectiveness failure rate. Add more only when these are stable.

3

Baseline From Existing Records

Most of these measures can be calculated backward from the deviation system, the CAPA system, and management review minutes. Build at least four quarters of history before anyone sees a trend line, so the first chart is not mistaken for a verdict.

4

Set No Targets for the First Two Quarters

Report the measures and discuss them, but do not attach goals. This gives the site time to see how the measures behave and lets the reporting rate rise, if it is going to, without being treated as failure.

5

Review the Layers Together in Management Review

Put all three layers on one page. The discussion should focus on disagreements between layers: outcomes improving while speak-up signals weaken, or leadership measures strong while staff say they never hear back. Every review should end with at least one decision tied to a layer one measure.

6

Show Staff What Changed

Publish a short summary of what leadership decided as a result of the scorecard. This is the step most programs skip, and it is the one that makes the next survey and the next quarter of reporting believable.

Who Sees What

Layer one measures belong in front of the leadership team, because they describe the leadership team. Layer two measures should go to area and shift leaders as well, since a long event-to-record lag on one shift is usually a supervisor conversation, not a site-wide issue. Layer three measures go everywhere, with the explicit message that they are lagging confirmation and not the place to look for this quarter’s actions. Never rank individuals or teams on reporting volume in either direction. Ranking teams that report the most encourages trivial reports. Ranking teams that report the least encourages silence.

How This Fits FDA’s QMM Program

FDA’s QMM program is voluntary and is not a CGMP inspection. In February 2026, FDA announced the third year of its prototype assessment protocol evaluation program, offering up to nine establishments the chance to participate and accepting requests through April 13, 2026.5 The notice says CDER evaluated nine establishments in 2024, found that the protocol and rubric distinguished maturity differences between practice areas at a single establishment and between establishments, and then streamlined the tool to make it clearer and more concise.5 Each assessment may take up to five days with a team of three CDER assessors, and participants receive a report with two to three strengths and two to three improvement opportunities in each practice area, followed by check-ins at about three and six months.5 FDA’s program page, last updated July 1, 2026, does not publish individual or aggregate scores.3

The white paper draws a distinction that matters for anyone building a culture scorecard. It describes quality metrics as quantitative data, and QMM assessments as qualitative and focused on approach, “including understanding why establishments select and define specific metrics for monitoring and how they leverage data to drive continual improvement.”4 In other words, a QMM assessor is less interested in the value of your metrics than in why you chose them and what you did with them. A scorecard built on the three layers, with written definitions, a documented rationale for each measure, and management review minutes showing decisions, is the kind of evidence that question calls for, whether or not the site ever volunteers for an assessment.

An illustration of one quarter’s review. Consider a hypothetical site whose layer three numbers look good: deviation volume down 15 percent, on-time closure up. Layer two shows minor deviation reports down more than majors, and median event-to-record lag on the night shift has doubled. Layer one shows two escalations from the quality unit waiting more than 60 days for a resource decision. The leadership team concludes that the falling deviation count is at least partly a reporting change, decides on the open resource requests that day, and schedules a listening session with the night shift. The decision is recorded, and the next quarter’s review starts with the follow-up. That sequence, not the numbers, is what quality culture measurement is for.

Conclusion

Quality culture is built by what leaders do, and it shows up first in whether people tell the truth about problems. Outcome metrics confirm it later. A scorecard that measures only outcomes asks leaders to steer by a number that moves a year after their decisions, and creates exactly the pressure to report less that a weak culture produces on its own. Sorting measures by how much leadership controls them fixes both problems: leaders see the results of their own behavior within weeks, speak-up signals show whether the organization believes it, and outcomes confirm whether the quality system is learning. Where the layers disagree, the disagreement is the most useful thing on the page.

Sakara Digital works with pharma and biotech organizations on quality systems, the data behind them, and the management practices that make them work. If you are rethinking how your organization measures quality culture, or want an independent view of whether your current metrics reward the behavior you intend, we are happy to have that conversation.

For Further Reading