Lab Instrument Data Integration: Raw Data, Metadata, and the Contextualization Gap

What the Instrument Actually Emits

Stand in front of a high performance liquid chromatography system in a quality control laboratory and watch what it produces. A detector signal over time. A set of integrated peaks with retention times and areas. A calculated concentration if the calibration is loaded. A timestamp. An instrument serial number if the acquisition software bothers to write one into the file header.

Now list what it does not produce. It does not know which batch of drug product the vial came from. It does not know whether the vial holds a release sample, a stability pull, or an investigational retest. It does not know that the method was revised three weeks ago and this is the first run under version 4. It does not know that the chemist started this run at 11:40 p.m. because the first attempt failed on a pump pressure alarm. It does not know that the second-person reviewer flagged an integration change and that the change was accepted.

None of that is a defect in the instrument. Instruments are built to measure. The problem is that regulators do not accept a measurement as a record. They accept a measurement bound to its context.

The regulation says this out loud

United States current good manufacturing practice regulation is unusually explicit here. 21 CFR 211.194(a) requires that laboratory records include complete data derived from all tests, and then enumerates eight specific items that must be present.¹ Read them as a list of context requirements rather than as a paperwork rule and the shape of the problem becomes obvious.

Of eight enumerated elements, exactly one is fully produced by the instrument without help, and even that one is incomplete because it is not identified to the lot tested. The other seven are joins to information held somewhere else. A laboratory that has “integrated its instruments” but only moves the result value has closed the easiest part of the gap and left the regulated part open.

Why it usually goes unnoticed

The gap hides because people fill it. A chemist knows the vial in position 14 is the stability pull because they loaded the tray. A supervisor knows the repeat run happened because they authorized it. A reviewer knows the method changed because they approved the change. The system does not know any of it, but the humans do, so the laboratory functions and the batches release.

The gap becomes visible in exactly three situations, and all three are expensive. It shows up during an inspection, when someone asks to trace a specific result back to its source and the trail runs through an uncontrolled spreadsheet. It shows up during an investigation, when you need to find every result produced on a given column or a given reagent lot and no field records which was used. And it shows up when someone tries to train a model or build a trending program on historical laboratory data and discovers that the results cannot be grouped reliably because the context that would group them was never written down in a machine-readable place.

Three Terms That Are Not Interchangeable

Raw data, metadata, and contextualized data get used as if they were synonyms in most laboratory conversations. Two of the three have precise regulatory definitions with different obligations attached. The third has no regulatory definition at all, which is exactly why it needs one inside your organization.

Raw data

The MHRA definition is the clearest one in circulation. Raw data is the original record, stored in the file or format in which it was originally generated, and it must permit the full reconstruction of the activities that produced it. MHRA also states that raw data is synonymous with source data as defined in ICH good clinical practice.³ PIC/S PI 041-1 defines it as the original record, described as the first capture of information.⁴

Two words in the MHRA definition carry all the weight. Format means the obligation attaches to the file as generated, not to a rendering of it. Reconstruction means the obligation is functional: the retained thing has to be enough to redo the work and reach the same conclusion. A printed chromatogram fails both tests. It is not the generated format, and it does not permit reintegration.

Metadata

FDA defines metadata plainly as the contextual information required to understand data, and notes that the term reaches beyond electronic records.² MHRA describes metadata as data that describe the attributes of other data and provide context and meaning, covering structure, data elements, interrelationships, and other characteristics.³ The EMA guideline on computerised systems and electronic data in clinical trials is the most direct of the three: metadata form an integral part of the original record, and without the context provided by metadata the data have no meaning.⁷

That last sentence is the regulatory basis for the entire argument in this article. Metadata is not documentation about a record. Metadata is part of the record. Losing it does not degrade the record. It destroys it.

Contextualized data

There is no regulatory definition of contextualized data, and that is not a gap in the regulation. Regulators define what must be retained. They do not define how you assemble it. Contextualization is the engineering answer to a compliance requirement, and because it has no external definition, most organizations never write one down.

A working definition that holds up in practice: contextualized data is a result that has been bound, in a machine-readable and durable way, to every element required to reconstruct and defend it, without depending on a person to supply the link. The phrase “without depending on a person” is the whole test. If a competent reviewer with no access to the people involved cannot reconstruct the event five years from now, the data is not contextualized, however well the laboratory currently functions.

Static and dynamic records

PIC/S makes a distinction that matters more in laboratories than anywhere else. A static record is fixed, such as a scanned image or a signed paper form. A dynamic record allows interaction: a chromatogram can be reintegrated, a spectrum can be baseline-corrected, a report can be recalculated with different parameters.⁴ The retention obligation for a dynamic record includes the ability to interact with it. This is the single most misunderstood point in laboratory data retention, and it is the point on which the proprietary format question turns.

Retention: What You Keep, for How Long, in What Form

Retention is where the three definitions above stop being academic. Different layers of the same result carry different obligations, and organizations routinely apply the loosest obligation to all of them.

Retention periods themselves vary by domain and by market, and they run from years past product expiry in a manufacturing context to decades in a clinical context. The specific number matters less than the planning horizon it implies. Long term preservation work in life sciences commonly assumes a fifteen year readability horizon or longer, which is well beyond the period any software vendor will commit to supporting a given version of a product.¹⁷ That mismatch is not a procurement failure. It is a structural feature of the problem, and it has to be designed for rather than negotiated away.

What a true copy has to preserve

Regulators do permit copies. MHRA is specific about what an accurate and complete copy has to include: the meaning of the data, covering items such as date formats, context, layout, electronic signatures and authorizations, and the full GXP audit trail.³ Where a document is electronically signed, the metadata associated with that signature must be retained as well.

Read that as a specification for any migration or archive project. If you move laboratory data out of a retiring system, the acceptance criteria are not “the values match.” They are: the values match, the units and formats mean the same thing, the audit trail came with them, the signatures are still attributable, and a dynamic record is still dynamic. Most migration plans test the first and assume the rest.

The draft Annex 11 direction

The European Commission and PIC/S published a substantially expanded draft of EU GMP Annex 11 for consultation on 7 July 2025, with the consultation period closing on 7 October 2025.⁵ The concept paper that preceded it set out the reasons for the revision, including the need to address the data life cycle and the increasing reliance on external service providers.⁶ The draft treats data integrity as a property to be designed into the system life cycle rather than checked at the end, and it expands what is expected around audit trails, periodic review, supplier oversight, and security.

For laboratory data specifically, the direction of travel is consistent across FDA, MHRA, PIC/S, and the Annex 11 draft: the obligation is attached to the data throughout its life, it survives the system that created it, and the organization is expected to be able to say who is accountable for it at each stage. That last point is the ownership question, and it is treated in its own section below.

Five Integration Patterns and Where Each Stops Scaling

Every laboratory in life sciences runs some combination of five patterns. They are usually described as a maturity ladder, with manual file drops at the bottom and a data platform at the top. That framing is misleading. Each pattern is the right answer for some part of the instrument estate, and each one has a specific point at which it stops working. What causes trouble is not running several patterns at once. It is not knowing which instruments sit in which pattern, and not having decided why.

Pattern 1: Manual file drop into a shared folder

The instrument writes its output to a local disk. Someone copies the file to a network share, names it according to a convention, and records the file name on a worksheet. This is still the most common pattern in the industry for stand-alone instruments, and it will be for years.

What it does well: nothing needs to be built, and it works with any instrument regardless of age or vendor. For a low-volume instrument producing a handful of results a month, it can be entirely defensible if the naming convention is controlled and the transfer is verified.

Where it fails: the join between the file and its context is a human act, recorded in a file name and a worksheet entry. File names are not audit-trailed. Copies are not verified unless someone verifies them. The original often stays on the acquisition workstation, so there are now two copies and no definition of which is the record. And the folder becomes unsearchable long before it becomes unmanageable.

Where it stops scaling: at roughly the point where one person can no longer remember the convention, or where any result needs to be found by an attribute other than its file name. In practice this is a few thousand files, or the first investigation that requires finding every run performed with a particular reagent lot.

Pattern 2: Direct instrument to LIMS interfacing

A driver or parser reads the instrument output and writes the result into a LIMS sample record. Sample identity flows outward via a worklist or a barcode, and the result flows back in. Done well, this closes several of the 211.194(a) joins automatically.

What it does well: it fixes the identity join, which is the single most valuable one. It removes transcription, which removes a class of error and a class of finding. When the LIMS holds the specification, it also closes the comparison-to-standard join.

Where it fails: point-to-point interfaces are built per instrument model, per software version, and often per method family. Each one is a small validated asset with its own change control obligation. A vendor software update can break a parser silently, and the failure mode is a value landing in the wrong field rather than an error message. The raw file usually stays behind on the instrument; the LIMS receives the result, not the record.

Where it stops scaling: at the point where the number of interfaces multiplied by the frequency of vendor updates exceeds what the validation and support model can absorb. Organizations tend to hit this between twenty and forty active interfaces, at which point the interface backlog stops moving and new instruments quietly default to Pattern 1. The related question of whether the underlying LIMS is worth extending is a separate decision, and one worth making explicitly rather than by accumulation.

Pattern 3: OPC UA for process and, increasingly, laboratory instruments

OPC UA is the established route for process instrumentation, where equipment publishes an information model that control systems, historians, and manufacturing execution systems can consume without a bespoke driver. ISPE has published concept work on using OPC UA alarms and conditions to carry pharmaceutical alarms and audit trail events from equipment to higher level systems, which is precisely the contextualization problem stated in process terms.¹³

The laboratory extension of this is newer. The OPC UA Laboratory and Analytical Device Standard, known as LADS, was released in January 2024 as an official companion specification. It models laboratory equipment through a hardware view and a functional view, and takes a device-agnostic approach so that a wide range of instrument types can be described consistently.¹¹¹²

What it does well: it moves semantics, not just values. An information model carries units, ranges, device state, and event structure, which means the receiving system does not have to guess. For manufacturing-adjacent laboratory equipment, it puts laboratory results on the same footing as process data.

Where it fails: only instruments that ship with a conforming server participate, and most of the installed base does not. Retrofitting is possible but is itself an integration project. It also does not solve the raw file problem: an information model is excellent at describing state and results, and is not where a full chromatographic data file lives.

Where it stops scaling: it does not stop scaling technically. It stops at the boundary of the installed base. Plan for it on new equipment purchases and on process instrumentation, and do not plan for it to cover a laboratory full of instruments bought between 2005 and 2020.

Pattern 4: Vendor middleware and instrument control platforms

A chromatography data system, a plate reader suite, or a vendor platform controls a family of instruments, holds the raw data, applies the method, and produces the result. Integration then means integrating with the platform, not with each instrument.

What it does well: it consolidates. One validated system, one audit trail, one user administration model, one set of interfaces to maintain instead of thirty. Within its own family, contextualization is often excellent: the system knows the method version, the calibration, the sequence, the user, and the reprocessing history.

Where it fails: the context is excellent inside the platform and largely invisible outside it. The interface out of the platform usually exposes results, not the full record. And the platform is where proprietary format risk concentrates, because the raw file it holds is typically a closed binary structure that only that software can open.

Where it stops scaling: at organizational boundaries. Two sites running two different vendors’ platforms produce results that cannot be compared without a translation layer, and the translation layer has to encode assumptions that neither vendor documented. The more successful the platform is inside its family, the harder the boundary becomes.

Pattern 5: A scientific data management system as a landing layer

Instead of pushing every instrument into the LIMS, the raw output lands in a system whose job is to capture, catalog, and preserve it, attach metadata, and make it retrievable. The LIMS still holds the sample and the result. The landing layer holds the record.

What it does well: it separates two concerns that are usually confused. Capture and preservation of the raw file is one problem. Sample management and result reporting is a different problem. When the landing layer takes the first, the LIMS interface set gets much simpler, and the raw file stops living on acquisition workstations. It also gives you a single place to apply retention policy and a single place to attach standardized metadata.

Where it fails: a landing layer that only stores files is a better-organized shared folder. The value is entirely in the metadata attachment and the joins, and those still have to be defined by someone who knows what context each result type requires. Organizations buy the platform and then discover the specification work was the hard part all along.

Where it stops scaling: it does not, provided the metadata model is maintained. It fails when the model is defined once during the implementation project and never revised as new instrument types arrive.

The Proprietary Format Question

A chromatography data system stores its raw data in a format the vendor designed and does not fully publish. The same is true of most spectroscopy platforms, many plate readers, and a good share of the analytical estate. Inside the retention period, that format has to remain readable and, because the record is dynamic, reprocessable.

Almost every organization knows this. Almost none of them have a position on it, because the risk does not become visible until the moment it becomes very expensive: a system reaches end of support, or a site closes, or a product is divested and the acquirer wants the underlying data rather than the reports.

Why deferral is the default

The reasoning is understandable. The current system works. The vendor is large and appears permanent. Nobody has ever needed to reprocess a ten year old chromatogram. And the alternative, exporting everything into an open format, raises an immediate question nobody wants to answer: what exactly is lost in the export, and who signs that it does not matter?

That question is answerable, and answering it is the work. Long term preservation practice in regulated life sciences is clear that data obsolescence is the central risk in archiving, that proprietary formats are the mechanism by which it materializes, and that the mitigation requires portable preservation formats, contractual exit terms, and a documented migration approach rather than an assumption of vendor continuity.¹⁷

The four available positions

Most organizations will end up with a mix: the open export as the durable layer, the original file retained alongside it, and a bounded period of legacy system availability during transition. What matters is that the position is written down per instrument family, with the reasoning, so that it can be defended and revisited.

Open formats are further along than most teams assume

The Allotrope Foundation, a consortium of pharmaceutical companies, instrument vendors, and software companies, has developed a set of standards for laboratory data, including an ontology, the Allotrope Data Format, and the Allotrope Simple Model. The Simple Model is a lighter-weight, community-driven schema intended to make harmonized data practical across instruments, teams, and sites, and work has continued on certification so that independent implementations produce consistent files.¹⁵

What changed recently is availability of open tooling. An open source library of converters that translate instrument output into the Simple Model schema is publicly maintained, which removes the argument that adopting the standard requires building every adapter yourself.¹⁴¹⁶ That does not make conversion free of judgment, and it does not remove the need to document what a conversion preserves. It does mean the open format position is now a practical option rather than a research topic.

Who Owns the Raw File

Ask three groups who owns a raw chromatographic data file and you will get three answers that do not overlap. The laboratory says IT owns it, because IT owns the server and the backup. IT says the laboratory owns it, because IT does not know what the file means or how long it must be kept. Quality says whoever generated it owns it, because quality’s role is oversight rather than custody. Everyone is being reasonable, and the file has no owner.

This is not a cultural problem to be solved with better collaboration. It is a definitional problem: “ownership” is being used for four different accountabilities that need to be assigned separately.

The inspection version of the ownership question

Regulators ask this question, though not in these words. In an August 2025 warning letter to a contract testing laboratory, FDA cited a failure of quality unit oversight, including the absence of restricted access and appropriate controls on computerized systems to prevent alteration or deletion of laboratory data, and the use of an uncontrolled document held by laboratory staff for sample preparation.¹⁹

The remediation FDA requested is the part worth reading closely. It asked for a list of all hardware, including both stand-alone and networked equipment in the laboratory; a list of all software configurations and versions; details of user privileges specifying roles and permissions for everyone with access, including administrative rights, with their organizational affiliation and title; and the oversight responsibilities for the computerized systems. That is an ownership map, requested as an inventory. An organization that cannot produce it on request does not have one.

The uncontrolled sample preparation document is worth noting separately, because it is the ownership gap in its purest form. Context that the record requires was real, was used daily, was written down, and lived entirely outside any controlled system. Nobody owned it, so nobody controlled it.

Stand-alone instruments are where ownership fails

Networked systems tend to acquire owners because they appear in an IT inventory and an application catalog. Stand-alone instruments do not. A balance with a local printer, a pH meter with a serial output, an older plate reader on a workstation under a bench: these are equipment to the laboratory and invisible to IT, and their data has no custodian at all.

A practical rule that resolves most of it: if a piece of equipment produces a GxP record, it has a named record owner and a named system owner, regardless of whether it is connected to a network. That single rule, applied through the equipment register rather than the application catalog, surfaces the stand-alone estate faster than any assessment.

A Target State for Contextualization

Here is the argument this article exists to make. Contextualization is treated as an integration chore because it is described as one: a backlog of interfaces, sized in effort, prioritized against other IT demand, and never finished. Described that way it cannot be finished, because there is no definition of done.

Treated as a data quality attribute, it acquires one. Every other quality attribute in a regulated laboratory has a specification, a measurement method, an acceptance criterion, and a review cadence. Contextualization can have all four.

The specification: a required context set per result type

For each result type the laboratory produces, the record owner defines the set of context elements the record must carry. This is a short document, not a project. For a release assay it might be: material identity, lot, sampling point, sample receipt date, specification version, method identifier and version, equipment identifier, column or consumable identity, calibration reference, operator identity, review identity, run start time, and the event that triggered the run.

Writing this down is the highest-value hour in the entire program, and it is almost always skipped, because everyone assumes it is obvious. It is not obvious. Two people in the same laboratory will produce different lists, and the difference between their lists is exactly the ambiguity that shows up in an investigation.

The measurement: contextualization completeness

Take a random sample of results of a given type from a defined period. For each one, attempt to assemble the full required context set using only machine-readable sources, with no interpretation and no asking anyone. Score the proportion of results for which every element is present and unambiguous. That number is contextualization completeness, and it is directly comparable across laboratories, sites, and years.

Two secondary measures make it more useful. Join reliability asks how many of the assembled links depend on a free-text field or a file naming convention rather than a controlled identifier. Reconstruction time asks how long it took to assemble the set, which is the operational proxy for what an investigation will feel like.

The acceptance criteria

Acceptance criteria have to be set by risk, not uniformly. A stability result supporting a registered shelf life and a routine in-process check do not warrant the same investment. A workable structure sets three tiers.

The point of the tiering is not the tiers. It is that once you have written them, an interface request stops being a preference and becomes a gap against a stated criterion. That changes who has to justify what, and it is the mechanism by which this work finally gets funded.

The FAIR overlap, stated carefully

The FAIR principles for scientific data management, published in 2016, set out that data should be findable, accessible, interoperable, and reusable, with an emphasis on machine-actionability: systems should be able to find, access, and reuse data with minimal human intervention.⁸ The overlap with the contextualization target is substantial, and it is worth naming because many research organizations already have a FAIR program running.

The overlap is not complete, and pretending otherwise causes problems. FAIR is oriented toward reuse and discovery. GxP contextualization is oriented toward reconstruction and defensibility. A dataset can be highly FAIR and still fail a data integrity expectation, because FAIR does not require an audit trail, does not require attributability to an identified individual, and does not require that a dynamic record remain reprocessable. Run the two programs together where they align, and keep the acceptance criteria separate.

Sequencing the Work

The most common way this work fails is starting with a platform selection. A landing layer is often the right answer, and it is never the right first step, because the specification work that makes it valuable has not been done and the platform will be configured against assumptions rather than requirements.

What to do about the platform decision

By the time phases one through three are complete, the platform question usually answers itself. If the inventory is small, concentrated in one or two vendor families, and the joins are mostly closable in the LIMS, a landing layer may be unnecessary. If the inventory is large and mixed, retention horizons are long, and multiple sites produce comparable results, a landing layer is close to unavoidable and you will now be able to specify it properly.

Either way the decision is made against measured gaps rather than a vendor’s description of the problem. That is worth the few months it takes to get there, and it is generally faster than the alternative, in which a platform is selected first and then spends eighteen months being configured against requirements that are still being discovered.

Conclusion

The contextualization gap is not an integration backlog that happens to be long. It is a quality attribute that has never been specified, so it has never been measured, so it has never been improved on purpose. Once you write down what context a result type requires, sample real results against it, and report the number, the work stops being a negotiation between the laboratory and IT about interface priority and becomes what it always was: a gap against a stated requirement, with an owner and a due date.

The four hard parts are each solvable, and none of them is primarily technical. Getting precise about raw data, metadata, and contextualized data prevents a retention plan built on the wrong object. Choosing an integration pattern deliberately per instrument, and knowing where each one stops scaling, prevents an estate of undocumented exceptions. Taking a written position on proprietary formats before a system reaches end of support keeps four options open instead of one. And separating record owner, system owner, data steward, and archive custodian resolves the ownership question that otherwise leaves the raw file with no custodian at all. The organizations that handle laboratory data well are not the ones with the newest platform. They are the ones that decided who is accountable for the file and wrote down what it has to contain.

Sakara Digital works with pharma and biotech organizations on data quality and laboratory data architecture, including contextualization target states, format and archive positions, and the ownership model that has to sit underneath both. If you are looking at an instrument estate you cannot fully account for, or a retention horizon that outlasts your systems, and you want an independent perspective on where to start, we are happy to have that conversation.

For Further Reading