Data Lineage Adoption in Clinical Data Management: Where Teams Stumble

Why CDM Lineage Is Different From Enterprise Lineage

Most enterprise data lineage conversations start with a data catalog vendor demo, a diagram of connectors to Snowflake and dbt and Tableau, and a promise that column-level lineage will emerge automatically from parsing SQL. That mental model does not travel well into clinical data management. The problems it solves are not the problems a clinical data manager, statistical programmer, or biostatistician actually has, and the tools it produces rarely see the transformations that matter most in a regulated trial.

Clinical data flows through a narrower pipeline than enterprise data, but a much higher-stakes one. A single EDC study database usually feeds one or a small number of downstream repositories, then a fixed set of SDTM domains, then a fixed set of ADaM datasets, then a bounded set of tables, listings, and figures that show up in a Clinical Study Report. There are very few branches. There are almost no ad-hoc consumers. Yet the transformations along that narrow path can determine whether a treatment effect reaches statistical significance, and whether a regulator can independently reconstruct the analysis five years after database lock¹.

Enterprise lineage tools were built to answer questions like “if I change this Salesforce field, what breaks downstream?” CDM lineage exists to answer a different question: “how do I prove that this specific value in a submitted analysis dataset came from a specific value on a specific case report form at a specific site on a specific date, and that every transformation in between was pre-specified, executed by a validated system, and reviewed by a qualified person?”² Those two questions require different metadata, different granularity, and a very different definition of “done.”

The EDC to CSR Flow: Where Lineage Actually Lives

Before diagnosing the pitfalls, it helps to be precise about what the CDM lineage chain actually looks like. Every clinical trial follows some version of the same flow, even if the labels vary by sponsor and CRO. Source data enters through an EDC platform (Medidata Rave, Veeva CDMS, Oracle Clinical, Castor, or one of a dozen others), gets aggregated and cleaned in a clinical data repository or data warehouse, gets restructured into CDISC SDTM domains, gets further derived into ADaM analysis datasets, gets rendered into tables, listings, and figures by statistical programmers, and finally gets narrated in the Clinical Study Report submitted to health authorities⁵.

Every one of those six steps has an audit trail somewhere. The question that separates mature CDM operations from immature ones is whether those audit trails compose into a single, queryable, machine-readable lineage graph, or whether they sit in six different systems in six different formats that a human has to stitch together during an inspection.

Five Places Teams Stumble

Across every CDM lineage assessment we have run, the same five failure patterns show up, in some combination, in nearly every organization. None of them are exotic. They are all consequences of treating lineage as a compliance artifact rather than an engineered capability.

1. Lineage that stops at the CDR boundary

The single most common failure. Lineage is captured meticulously inside the EDC (because the EDC vendor built the audit trail) and inside the CDR (because the reconciliation platform logs every change). Then a statistical programmer runs a SAS program that transforms the CDR into SDTM, and the lineage evaporates. The Define-XML says the SDTM.LB.LBORRES variable “originates from source data” and cites a CRF page, but there is no machine-readable link between the specific record in SDTM.LB and the specific CDR row it was derived from. If a reviewer wants to trace a lab abnormality flagged in the safety narrative back to a specific site visit, someone has to open a SAS program and read the code¹.

This is not a technology problem. It is a scope problem. The lineage tool the enterprise data team picked (Collibra, Alation, or one of the modern open-source options) was built to parse SQL against a data warehouse. It does not read SAS. It does not understand CDISC. It does not know that ADSL.TRT01P is derived from EX.EXTRT and DM.ARM, or that the derivation rule changed between v1.0 and v2.1 of the ADaM specification.

2. Missing transformation logic capture

The Pistoia Alliance and every regulator that has written on this topic have converged on the same requirement: lineage must include not just where data moved but how it was transformed⁸. Yet in most CDM organizations, the transformation logic sits inside SAS macros, R functions, or Python scripts stored in version control (if you are lucky) or on a shared drive (if you are not). The Define-XML captures a human-readable description of the derivation, but the description almost always oversimplifies what the code actually does. When code and prose disagree, code wins, and prose is what the reviewer sees.

Sanofi has written publicly about a pseudo-code standard specifically to close this gap: a structured, machine-readable representation of derivation logic that sits alongside the executable code and stays in sync with the Define-XML⁹. That is the direction the industry needs to move. Most teams are still years away.

3. Orphaned derived variables

ADaM datasets accumulate derived variables across the life of a compound. A variable added for a secondary endpoint in Study 302 gets copied into the ADaM specification for Study 303, then subtly modified for Study 401, then repurposed for an integrated summary of safety at NDA time. Somewhere along the way the derivation rule diverges from the original. Or a variable is added by one programmer, never used in a TLF, and quietly forgotten. Or a variable that was retired in a spec update remains in the actual dataset because nobody ran the cleanup.

4. Insufficient granularity for regulatory reconstruction

Lineage at the dataset level is not lineage. It tells a reviewer that ADSL was derived from DM, EX, and DS. It does not tell them how a specific value of ADSL.TRTSDT for a specific subject was computed. The FDA and EMA expectation is not that you can describe the flow at a high level; it is that a reviewer can, given a specific value in a submitted dataset, reconstruct the derivation without contacting the sponsor².

Record-level lineage is expensive. Column-level lineage is table stakes. Most CDM organizations we assess have neither. They have process-level lineage documented in an SOP and a data flow diagram, plus dataset-level metadata in the Define-XML. When a reviewer asks a specific-value question, the answer requires a programmer.

5. Query and reconciliation history that does not join

Every discrepancy managed through the query workflow is a mini-transformation of the source data. A CRF field was entered as one value, the DM raised a query, the site responded with a different value, and the CDR now holds the updated version. That query history is captured in the EDC and the CDR, but it is rarely linked to the SDTM record that carries the final value. When a reviewer or auditor traces a value back through SDTM to the CDR, they see the current value; the fact that this value was the resolution of a query that changed the answer from 14 to 41 is buried in a separate system¹⁰.

The same problem shows up with medical coding. A free-text adverse event term is coded to a MedDRA preferred term through a coding workflow that involves auto-suggestion, medical review, and sometimes revision after data cleaning discussions. The final coded term appears in SDTM.AE.AEDECOD, but the sequence of coder decisions that produced it typically sits in the coding tool’s audit log, not linked to the SDTM record. When a safety reviewer asks why a specific adverse event was coded to one preferred term rather than a similar one, the answer requires a coder to open the coding tool and reconstruct the history from a separate audit trail.

The consequence of all five failure patterns is the same: lineage that is technically compliant on paper but functionally inadequate when a regulator, an internal auditor, or a new statistical programmer needs to answer a specific-value question. In every case the sponsor eventually gets there, but the path involves phone calls, code review, and time that a well-instrumented operation would not need to spend.

A Five-Level Maturity Model for CDM Data Lineage

Maturity models risk becoming decorative if they are not tied to observable behavior. The version below is calibrated to what we have actually seen work and fail in mid-size and large pharma CDM organizations. It is regulatory-first: each level is defined by what the organization can demonstrate to an inspector in an unannounced review, not by what tools it has bought.

Most large sponsors we work with sit at Level 2 or Level 3. Many CROs sit at Level 2. A handful of the most digitally mature sponsors are pushing into Level 4 for their flagship compounds, often with different maturity levels for different therapeutic areas or acquired legacy programs. Level 5 is rare and typically requires a multi-year platform investment tied to a broader clinical data science transformation¹¹.

What the levels look like in practice

It is worth being concrete about what each level looks like when you walk into an operational review, because the model only helps if you can recognize where you are.

Level 1 (Tribal) shows up as a study team that answers every lineage question with “let me ask Dave.” Dave is the senior programmer who built the ADaM specifications ten years ago and has stayed with the compound through three protocol amendments. Dave is a treasure. Dave is also the single point of failure for every regulatory response, every integrated summary, and every question a new therapeutic area medical lead asks about how the analysis works. When Dave retires, the compound’s entire lineage capability walks out with him.

Level 2 (Documented) shows up as a well-organized network share with data flow diagrams, DMPs, SAPs, ADaM specifications, and Reviewer’s Guides. Everything the reviewer needs is technically in there. Finding it takes hours. Answering a specific-value question still requires opening the SAS code. This is where most sponsors sit for most compounds.

Level 3 (Metadata-Driven) shows up as an MDR that everyone in CDM, biostatistics, and statistical programming actually uses. Define-XML is generated from the MDR, not authored separately. When a new study starts, the team pulls dataset and variable definitions from the MDR and inherits controlled terminology automatically. Column-level lineage questions can be answered from the MDR without opening code, though record-level questions still require a programmer.

Level 4 (Executable Lineage) shows up as a standards library of reusable, machine-readable derivations. When a programmer writes a new derivation, they express it in a structured form that generates both the executable code and the Define-XML entry. Query history is joined to SDTM. A senior programmer can answer a record-level lineage question by running a query, not by reading code. This level requires sustained investment and is rare in industry.

Level 5 (Continuous Lineage) shows up as an event-driven pipeline where every execution updates the lineage graph. Anomaly detection runs continuously and surfaces orphaned variables, unexpected derivation drift, or missing metadata. Change impact analysis runs before a derivation rule is updated. Almost no CDM organizations are here yet; the ones approaching this level are typically the ones that have built or bought a modern clinical data science platform and staffed it with data engineering talent.

Level transitions are not linear

Moving from Level 2 to Level 3 is a metadata investment: buying or building an MDR, migrating from Define-XML authoring to Define-XML generation, and training programmers to work with the MDR as the source of truth. It is largely a documentation and tooling shift.

Moving from Level 3 to Level 4 is a fundamentally harder step. It requires the organization to commit to a machine-readable representation of derivation logic and to enforce, through governance, that every new derivation is expressed in that representation. This is where most transformation efforts stall, because the incremental cost of writing pseudo-code alongside SAS is real, and the benefit accrues to future studies and future inspectors, not to the programmer writing it today.

Moving from Level 4 to Level 5 is an infrastructure investment: pipeline orchestration, event-driven metadata capture, and the analytics layer needed to surface anomalies. By this point the organization has usually rebranded from Clinical Data Management to Clinical Data Science and is running a fundamentally different operating model¹².

Organizational Patterns That Work Versus Fail

Tools and frameworks are necessary but not sufficient. Two organizations can buy the same MDR platform and end up at different maturity levels three years later, because the organizational patterns around the tool determine whether it becomes a system of record or a shelf-ware compliance artifact. Here are the patterns we see separating success from failure.

Reading the Regulatory Signal

Regulators have been signaling for years that they expect more from clinical data lineage, and the signal has sharpened in the past three years. Sponsors who are still treating lineage as a documentation exercise are working from a five-year-old model of what an inspection looks like.

The FDA’s 21 CFR Part 11 has always required trustworthy, reliable, and equivalent electronic records with a robust audit trail¹³. The EMA published its final guideline on computerised systems and electronic data in clinical trials in 2023, replacing the 2010 reflection paper and codifying ALCOA+ (attributable, legible, contemporaneous, original, accurate, plus complete, consistent, enduring, and available) with a new emphasis on the Traceable principle. That new principle explicitly reflects the reality that data now moves across many interconnected systems, and that traceability must survive those transitions¹⁴.

ICH E6(R3) reinforces the same theme with a risk-proportionate approach to computerised systems and data lifecycle. The direction of travel is unambiguous: regulators are moving from “show me your audit trail” to “show me the complete lineage from source to submission, and let me query it directly.”

The regulatory signal is not the only pressure. The 2023 AstraZeneca warning letter and the ongoing series of data integrity findings across the industry have made clear that inspectors are increasingly willing to translate lineage gaps into 483 observations. And the emergence of AI-assisted analyses in submission dossiers is pushing the bar higher still: if a machine learning model contributed to a decision, the lineage of the training data, the features, and the model outputs all now sit inside the regulatory scope¹⁵.

Getting Started Without Boiling the Ocean

The most common mistake we see is the opposite of stagnation. It is the enterprise-scale, three-year, cross-therapeutic-area lineage program that consumes budget for eighteen months and produces a governance council but no working lineage. Programs that succeed almost always start narrower, run faster, and use one high-visibility study or compound as the anchor.

What not to do

Do not try to solve CDM lineage with an enterprise data catalog alone. Do not treat lineage as a quality-owned initiative. Do not build a lineage program that has no upcoming submission attached to it. Do not fund the tooling and neglect the standards work. Do not confuse “we have an audit trail in the EDC” with “we have lineage.” And do not build the program in a way that would not survive the departure of your lead statistical programmer, because that is exactly the risk you were trying to reduce.

The role of AI, and what it does not solve

Every CDM leadership team we speak with in 2026 is under pressure to explain how AI fits into the data lineage story. The honest answer is nuanced. AI-assisted mapping of legacy CRF data to SDTM is real and increasingly capable, and metadata repositories are beginning to auto-suggest domain assignments and controlled terminology mappings¹². Large language models can accelerate the drafting of Reviewer’s Guides, the review of Define-XML entries, and the identification of specification-to-code discrepancies. Anomaly detection over the pipeline can surface orphaned derived variables and derivation drift that a human review would miss.

What AI does not solve is the underlying capability gap. A team at Level 2 that adds AI-assisted mapping is still at Level 2, because the mapping output still lands in the same manually-authored artifacts and the same specification-code disconnect persists. AI amplifies the maturity level you already have; it does not replace the metadata infrastructure required to move up a level. The organizations that will get the most out of AI-assisted CDM in the next three years are the ones that have already invested in the MDR, the standards library, and the machine-readable derivation logic that gives AI something to reason about.

There is also a regulatory dimension. Any AI-assisted step in the pipeline needs its own lineage: what model was used, what version, what training data, what human review, what decision was recorded. Adding AI without extending lineage to cover it moves the organization backwards, not forwards. Sponsors that have integrated AI thoughtfully treat the AI itself as a data producer with its own lineage requirements, not as an invisible transformation that happens outside the traceability chain.

Conclusion

Data lineage in clinical data management is a narrower, deeper, and more regulated problem than the enterprise data lineage that most vendors sell against. It lives in the specific chain from EDC through CDR through SDTM through ADaM through TLF to CSR, and its purpose is not analytical curiosity or governance theater. Its purpose is to let a regulator, an inspector, or a future statistician reconstruct a specific value in a submitted analysis dataset without picking up the phone. Teams that treat it as anything else, whether as a compliance artifact, an enterprise IT deliverable, or a byproduct of buying the right catalog, end up with lineage that stops at the CDR boundary and a submission package held together by the memories of two or three senior programmers.

The organizations we have seen do this well have three things in common: they treat CDM lineage as an engineering capability, not a document; they invest in metadata infrastructure before they invest in fancy tooling; and they pick a level on a maturity model, publish it, and hold the organization to it. None of that requires a moonshot. It does require deciding that lineage is worth the same operational discipline as query resolution or database lock, and staffing it accordingly.

Sakara Digital works with pharma and biotech organizations building the CDM and clinical data science capabilities that support this kind of end-to-end lineage. If you are trying to move up a maturity level, thinking about the right sequencing of MDR investment and standards work, or preparing for an inspection where lineage is likely to be a focus area, we are happy to have that conversation.