Data Quality Gates in Validated Pipelines: CI/CD Checks That Satisfy QA

Two Teams, Two Vocabularies, One Set of Data

Walk into a data platform team in a mid-sized biotech and you will find something that would have been unusual in a regulated environment ten years ago. Transformation logic lives in a Git repository. Every change goes through a pull request with a required reviewer. A continuous integration job runs on every commit, executing hundreds of assertions against a test dataset. Deployment to production is automated. Nobody prints anything.

Walk into the quality organization in the same company and ask what evidence exists that the reporting dataset feeding the annual product quality review is trustworthy. You will be handed a validation package built around a different model: a validation plan, a requirements traceability matrix, executed test scripts with initialed and dated results, a summary report, and a signature page. The package was assembled once, at go-live, and refreshed when a formal change was raised.

Neither group is wrong. The engineering group is testing far more thoroughly and far more often than the validation package suggests. The quality group is asking for something the engineering group already has, expressed in a form the engineering group does not naturally produce. What is missing is a deliberate mapping between the two.

The mismatch is about form, not rigor

The interesting part is that regulators have already moved toward the engineering side of this argument. FDA’s guidance on computer software assurance for production and quality management system software, currently in a version issued February 3, 2026 that supersedes the September 24, 2025 final,¹² explicitly recommends using digital records rather than paper artifacts. The guidance recommends incorporating the use of digital records, such as system logs, audit trails, and other data generated and maintained by the software, instead of paper documentation, screenshots, or duplicating results already retained digitally by the software.¹

That single sentence resolves most of the argument, if teams read it. A continuous integration job log is exactly the kind of digital record the guidance describes. The problem is that a raw job log is not yet a validation record, because it does not answer the questions a reviewer asks. It usually does not state the intended use of what was tested. It rarely records the risk basis for why those particular checks were selected. It often does not contain a conclusion about acceptability, and it may not identify who reviewed it or whether anyone did.

The same guidance sets out what the record should contain: the intended use of the software feature, function, or operation; the outcome of the risk-based assessment; documentation of the assurance activities including a description of the testing performed, issues found such as deviations, defects, or failures, and a conclusion statement declaring acceptability of the software for its intended use; a record of who performed the testing and the date; and established review and approval where appropriate.¹

Read that list against a typical pipeline run and the gap is narrow. Most of it is metadata the pipeline could emit if someone asked it to. The engineering team never asked because nobody told them what the record needed to contain.

Why this matters more now than it did

Three things changed at once. Pipelines got faster, so the number of releases per year went from a handful to hundreds. Data volumes grew enough that manual review of loaded data stopped being possible. And the outputs of these pipelines started feeding decisions that regulators care about: batch release support, stability trending, deviation trending, comparability assessments, and increasingly, models that inform process control.

The draft revision of Annex 11 published for comment on 7 July 2025 runs to roughly nineteen pages against the four or five of the version it replaces, with substantially expanded treatment of audit trails, periodic review, supplier and service management, and security.⁶ Whatever the final text says, the direction is more specific expectations about computerised systems, not fewer. A data platform that cannot explain how it controls its own data quality checks is going to be a harder conversation in 2027 than it was in 2022.

Build Time and Run Time Gates Are Different Controls

If you take one structural idea from this article, take this one. A data quality gate that runs before code is released and a data quality gate that runs on production data during a load are not variations of the same control. They test different things, they fail for different reasons, and they carry different obligations.

Teams conflate them because in a modern stack they are written in the same language, live in the same repository, and run through the same orchestrator. A dbt test looks identical whether it runs in a pull request check or in the nightly production job. Great Expectations checkpoints run the same way in both places.¹⁴ That surface similarity is exactly what causes the trouble.

What each one is actually testing

A build time gate tests the code. It asks whether the transformation logic, the schema definition, and the join keys behave as specified when run against known inputs. The subject under test is the software. If a build time gate fails, the code is not correct and should not be released. Nothing has happened to production data, because nothing has been released.

A run time gate tests the data. It asks whether the records that arrived in this particular load look like the records the process is supposed to produce. The subject under test is the data itself, and often, indirectly, the upstream process that generated it. If a run time gate fails, the code may be perfectly correct. Something happened in the world: an instrument was recalibrated, a site changed a coding practice, a source system was upgraded, a batch genuinely ran differently.

The consequence for your validation approach

Build time gates belong inside your software assurance approach. They are testing activities. Their evidence supports a release decision, and the release decision is a change control event that already has a home in your quality system.

Run time gates are something else. They are ongoing process controls, closer in character to an in-process check on a manufacturing line than to a test script. Their evidence supports a statement about a specific dataset on a specific day. Treating them as validation tests produces two failure modes at once: engineering gets buried in paperwork for routine data variation, and quality gets flooded with records that carry no useful signal.

Designing Build Time Gates That Produce Validation Evidence

Build time gates are the easier half, because the software industry has already solved most of the mechanics. The work in a regulated setting is deciding which of them carry validation weight, and making sure those emit a record that can stand on its own.

The three build time gates that earn their place

Schema contracts deserve more attention than they get

A schema contract is an explicit agreement between the team producing data and the teams consuming it, covering field names, types, nullability, and meaning. In distributed systems, that agreement is usually implicit: a table, a topic, an API response that everyone has quietly assumed will keep its shape. When the assumption drifts, pipelines break or, worse, keep running and produce wrong output.

In a GxP setting the schema contract is doing something beyond engineering hygiene. It is the concrete expression of what the data means, which is the same thing a reviewer needs to establish when asking whether a record is accurate and complete. MHRA’s data integrity guidance defines metadata as data that describe the attributes of other data and give them context and meaning, including structure, elements, and interrelationships.⁴ A versioned schema contract is metadata in exactly that sense, and it happens to be machine-checkable.

Practically, that means declaring the contract in the repository, versioning it alongside the code, and failing the build when a source violates it. It also means treating a change to the contract as a change to a specification, which is the point at which change control enters. More on that below.

What becomes a controlled record and what does not

This is the question that most often goes unanswered, and answering it explicitly removes a great deal of friction.

Writing this table down for your own environment, agreeing it with quality, and putting it in the procedure is perhaps the single highest-return hour of work in this whole topic. It ends the recurring argument about whether a given log needs to be retained, and it stops the quiet expansion of controlled records into everything the pipeline touches.

Designing Run Time Gates That Survive Production Data

Run time gates are harder, because production data is not obligated to behave. The design problem is picking checks that catch real problems without firing constantly on normal variation.

The three families that matter

Volume and completeness checks. Row count against an expected range, presence of all expected source partitions, freshness of the most recent record, and reconciliation of counts between the source and the loaded table. These catch the most common and most damaging failure in practice, which is a partial load that looks successful. A file that transferred half way, a site that did not submit, an API that paginated wrong: the pipeline reports green, the dataset is quietly incomplete, and the deficiency is invisible unless someone counts.

Field-level quality checks. Null rate against a threshold, values outside an allowed set, out-of-range numeric values, malformed identifiers, and dates that fall outside a plausible window. These catch upstream process changes and data entry problems. They are the checks most likely to be noisy, and the most likely to need a range rather than an absolute rule.

Distribution checks. Comparing the statistical shape of this load to a recent baseline: mean and spread of a continuous measure, the mix of categorical values, the proportion of records by site or by product. These are the checks that catch the subtle, slow problems, and they are also the checks most likely to be misinterpreted. A shift in distribution is information, not necessarily a defect.

Severity is a design decision, not a default

Most modern tooling already supports what is needed here. In dbt, a data test has a severity setting of either error or warning, and thresholds can be set with error_if and warn_if conditions so that a small number of failing records raises a warning while a larger number raises an error.¹²¹³ Great Expectations produces validation results for every checkpoint run and can trigger configured actions when those results come back.¹⁴¹⁵ Soda expresses checks in a declarative check language against defined datasets.¹⁶ Published work on multi-layer testing frameworks for cloud-native pipelines describes the same layered pattern of structural checks enforced strictly and statistical checks enforced with more tolerance.¹⁹

The tooling is not the constraint. The constraint is that somebody has to decide, for each rule, what happens when it trips, and that decision has to be written down before the rule goes live.

Three enforcement levels, defined up front

Every rule gets exactly one of these three levels, and the level is recorded in the rule definition, in the repository, alongside the rule. That makes the level itself version controlled, reviewable, and reconstructable later, which matters when someone asks whether a check was blocking at the time a particular load ran.

When a Run Time Gate Fails: Build Failure, Deviation, or Neither

This is the argument teams actually have, usually at nine in the evening with a batch release waiting. A distribution check tripped on the overnight load. Engineering says the pipeline worked exactly as designed and detected something, so this is not a failure of the system. Quality says a control fired on GxP data, so this is a deviation. Both positions are defensible, which is why the argument never resolves cleanly in the moment.

The answer is that neither position should be argued in the moment. The disposition should already be written in the procedure, keyed to the enforcement level and the nature of the check, decided when the rule was designed and approved.

The disposition matrix

Writing the procedure so the decision is already made

The procedural language that makes this work is short. For each data quality rule applied to a GxP dataset, the rule definition records the enforcement level, the acceptance criteria, the named role responsible for disposition, the period within which disposition must occur, and the trend rule if one applies. The procedure then states that a rule operating within its defined enforcement level is the system working as intended and does not, by itself, constitute a deviation.

That last clause is the one that saves the most argument. It says out loud that detection is the purpose of the control, and that detecting something is not the same as failing.

Where quality risk management thinking helps

ICH Q9(R1) makes the point that the level of effort, formality, and documentation of the quality risk management process should be commensurate with the level of risk.³ The revision specifically addressed formality in quality risk management and risk-based decision making as areas that needed clarification, because organizations were applying uniform heavy process regardless of what was at stake.

Applied here, that means a dataset that supports batch release warrants blocking gates, tight thresholds, and a short disposition window. A dataset that supports internal operational reporting warrants monitoring gates and a weekly review. The rules can be written in the same language, run in the same pipeline, and carry completely different consequences, and that difference should be traceable to a documented risk assessment rather than to who happened to configure the job.

Change Control on the Rules Themselves

Here is the point that surprises data teams. A data quality rule is a control. If it appears in a validation package or supports a statement about data used in regulated decisions, then changing it changes the level of assurance. Widening a null rate threshold from two percent to five percent is a change to a validated control, and it cannot be a pull request that one engineer approves on a Tuesday afternoon.

Equally, if every threshold tweak requires a formal change record with a quality signature, the team will stop tuning thresholds. Rules will stay badly calibrated, alerts will be ignored, and the control environment will degrade while looking fine on paper. Both extremes fail.

Tiering rule changes

The way out is to classify rule changes by what they do to the level of assurance, and to define the approval path for each tier in advance.

Making the tier decision enforceable rather than aspirational

A tiering policy written in a procedure and ignored in the repository is not a control. Three mechanisms make it real.

Separate the rule definitions from the transformation code. If quality rules live in their own directory or their own repository, you can apply different review requirements to them. Most Git hosting platforms support path-based code owner rules, which means a change touching the rules directory can automatically require an additional named approver.

Make the automated check detect the direction of change. A comparison between the current and proposed rule configuration can determine whether a threshold moved up or down, whether an enforcement level dropped, and whether a rule was removed. That comparison can post its result on the pull request and block merging until the corresponding change record is referenced. This is not exotic engineering. It is a script that reads two configuration files.

Require the change record identifier in the commit. For Tier 1 changes, the merge is blocked until the commit message or pull request references an approved change record. That single link is what lets a reviewer later trace a threshold value back to the assessment that justified it.

Periodic review of the rule set

Rules decay. Sources change, products retire, processes improve, and a check that was well calibrated two years ago now fires on every load or never fires at all. A periodic review, annually for most datasets and more often for high-risk ones, should look at each rule and ask three questions: has it fired in the review period, was every firing a true finding, and does the threshold still match the process. Rules that never fire may be redundant or may be scoped to nothing. Rules that always fire are not controls, they are noise. Both cases should result in a documented decision rather than quiet tolerance.

Evidence Retention: What the Pipeline Has to Emit

Now the practical test. Three years from now, an inspector points at a figure in a report and asks what data it came from and how you know that data was good. Somebody has to reconstruct the answer. Everything in this section exists to make that reconstruction possible.

The reconstruction test

Ask your team to answer this question about a load from eighteen months ago, using only what the systems retained. Which version of which rule ran against which data, what did it conclude, who reviewed the result, and what was decided?

Most teams cannot answer it on the first attempt. The job log was retained for ninety days. The rule definitions are in Git, but nothing recorded which commit was deployed at the time. The results were written to a monitoring dashboard that only keeps thirteen months. The review happened in a chat thread.

The minimum emission set

For every run of a run time gate against a GxP dataset, the pipeline should write a structured record containing the following. This is a short list, and every item on it exists because a reviewer will ask for it.

This record is the validation evidence. It is machine-generated, structured, queryable, and it contains everything the CSA guidance describes as belonging in the record: what was tested, what the risk basis was, what was found, what was concluded, who performed it, and when.¹ It is also far better evidence than a screenshot, because it can be aggregated, trended, and reconciled against the approved rule inventory.

Retention, immutability, and the audit trail question

Where these records live matters as much as what they contain. Three requirements apply.

Retention has to match the retention of the data it describes. If a dataset supports records that must be kept under 21 CFR 211.180, the quality evidence about that dataset has to be available for the same period.⁸ A ninety day log retention policy on the CI system is fine for the console output, and not fine for the structured gate results. Those two things need different homes and different retention rules, which is another reason the controlled record versus tooling distinction matters.

The records have to be protected from alteration. Part 11 requires secure, computer-generated, time-stamped audit trails that independently record the date and time of entries and actions that create, modify, or delete electronic records, and requires that audit trail documentation be retained for at least as long as the subject records.⁷ Gate results stored in a table that engineers can update are not going to satisfy that expectation. Append-only storage, or storage with an audit trail over it, is the design choice here.

The rule definitions have to be retrievable, not just versioned. Git history is a genuine advantage, and it is not automatically sufficient. Repositories get migrated, history gets rewritten, and organizations change hosting providers. If the retention period is longer than your confidence in the repository, periodically export the approved rule set as an immutable snapshot into the same document management system that holds your other controlled records.

Do not forget the passing runs

Teams frequently retain failures and discard successes, because failures are what generate work. That is backwards for validation purposes. The reviewer’s question is whether the control was operating, and the evidence that it was operating is the population of runs where it executed and passed. A record showing four failures in three years, with no evidence of the eleven hundred passing runs in between, tells a reviewer very little and invites the follow-up question you do not want.

Keeping It Proportionate Without Losing the Record

Everything described so far can be scaled up into a program that consumes more effort than the risk warrants. The discipline that prevents that is the same risk-based thinking that runs through current regulatory guidance, applied honestly rather than as a label.

Risk sits in the use of the data, not in the pipeline

The most common error in scoping is treating the pipeline as the unit of risk. Pipelines are usually mixed. The same platform often carries a dataset supporting batch disposition and a dataset supporting a departmental dashboard. Applying uniform controls to both means either over-controlling the dashboard or under-controlling the disposition data, and in practice it produces the first.

Scope by dataset and by use. For each dataset, record what regulated decisions depend on it, what happens if the data is wrong and nobody notices, and how quickly an error would be detected by another means. That assessment drives the enforcement levels, the thresholds, the disposition window, and the retention period. It is also the documented risk basis the record is supposed to contain.¹

Three questions that right-size the effort

What CSA thinking does and does not settle

Risk-based software assurance is the reason this approach is defensible rather than a shortcut. The guidance is clear that documentation of assurance activities need not include more evidence than necessary to show that the software performs as intended for the risk identified, and that digital records generated by the software are the preferred form of that evidence.¹ GAMP 5 Second Edition moved in the same direction, adding explicit treatment of critical thinking and of iterative software development approaches.¹⁸ FDA’s long-standing position on Part 11 scope has similarly emphasized applying controls based on a justified assessment of risk to record integrity rather than uniformly.¹¹

What none of that settles is the run time question. Software assurance guidance is about establishing confidence in the software. It says relatively little about what to do when correct software correctly reports that today’s production data looks different from yesterday’s. That gap is where your own procedure has to do the work, and it is why the disposition matrix described earlier is not something you can lift from a guidance document. It has to be written for your data, your processes, and your risk tolerance, and approved before you need it.

A Rollout Sequence That Does Not Stall the Data Team

The approach fails most often in adoption, not in design. Teams try to apply it to everything at once, the data group experiences it as bureaucracy landing on top of work that was going fine, and the effort stalls. A staged sequence works better, and the order matters.

Conclusion

The tension between modern data engineering and pharmaceutical validation is real, and it is not primarily a technology problem. Both groups are producing evidence. They are producing it in forms the other group does not recognize, and the reconciliation is being attempted after the fact instead of designed in. The fix is a set of decisions that take a few focused sessions to make and then hold for years: which gates run before release and which run on production data, which artifacts become controlled records, what each failure classification means, how a rule change gets approved, and what the pipeline must write down so that the record can be reconstructed later.

Two of those decisions carry most of the weight. The first is separating build time gates from run time gates, because that separation is what stops a normal data variation from being treated as a software failure and stops a software defect from being explained away as data noise. The second is deciding the disposition of a failed run time gate in advance, in writing, at the moment the rule is designed. Decisions made calmly during design are almost always better than decisions made at nine in the evening with a release waiting, and having them already made is what allows a data team to move quickly without the quality organization losing sleep.

Sakara Digital works with pharma and biotech organizations building validated data platforms and the quality processes that have to sit alongside them. If you are putting automated data quality gates into a regulated pipeline and want an independent perspective on where the controlled record boundary should sit, we are happy to have that conversation.

For Further Reading