Data Historian Modernization on the Pharma Shop Floor

Why the Historian Is the Hardest System in the Plant to Replace

Ask a manufacturing IT team to rank their systems by business risk and the historian rarely comes first. The MES is more visible. The ERP touches money. The LIMS blocks release. The historian just runs, quietly collecting tag values from distributed control systems, programmable logic controllers, building management systems, and standalone skids, and it has usually been running that way since before anyone currently on the team joined the company.

That is exactly the problem. A system nobody has needed to think about for fifteen years is a system whose behavior nobody has documented for fifteen years. And unlike most plant systems, the historian’s value is almost entirely retrospective. Its purpose is to answer questions that have not been asked yet, about batches that have already been released, using data that was collected before anyone knew the question existed.

What the historian actually holds

A mature site historian typically holds tens of thousands of tags. Temperatures, pressures, flow rates, pH, dissolved oxygen, agitation speed, valve positions, motor currents, differential pressures across filters, room classifications, chilled water supply temperatures, and every alarm and event that the control system generated along the way. Some of those tags map directly to critical process parameters named in the validated control strategy. Most do not. The ratio is often ten or twenty to one.

What makes this valuable is not the individual tag. It is the accumulated history. When a site needs to demonstrate that a process has remained in a state of control across three years of commercial production, the evidence comes from that archive. When an out-of-specification result triggers an investigation, the first question is almost always what the process was doing at the time, and the answer comes from the historian. When a technology transfer team needs to understand why a step behaves differently at a receiving site, they compare historian traces.

The dependency nobody drew on the architecture diagram

The FDA’s process validation guidance defines a three-stage lifecycle, with Stage 3 continued process verification described as an ongoing program of collecting and analyzing product and process data to confirm the process remains in a state of control.¹ In practice, the data that feeds Stage 3 at a commercial site comes from the historian. Statistical process control charts on critical process parameters, capability indices, trending of in-process measurements against action and alert limits: all of it is built on historized time-series data, usually pulled on a schedule into a reporting layer.¹⁵

That creates a dependency that rarely appears on the system inventory in the form it deserves. The historian is not a supporting system for CPV. It is the record. If a historian migration breaks the continuity of that record, the site has a validation problem, not an IT problem.

Three things that make it different from an ERP or LIMS replacement

Every regulated system migration is difficult. The historian adds three problems that most others do not have.

The record was transformed on the way in. A LIMS result is stored as it was reported. A batch record entry is stored as it was entered. A historian value has usually passed through an exception filter at the interface and a compression algorithm at the archive, both of which discard values on purpose. What sits in the archive is a reduced representation of what the instrument produced. That is a design feature, and it is defensible, but it has to be understood before anyone starts talking about a faithful migration.

The data is only meaningful in combination. A single tag value with a timestamp is close to useless. It becomes evidence only when you know which batch was running, which product, on which equipment train, in which phase of which unit procedure, with which material lots charged. That context usually lives in the MES, the batch execution system, or the ERP, not in the historian.

The system spans a security boundary. The historian collects from the control network and serves to the business network. It is one of the few systems that has to exist on both sides of the line, which makes any change to it a change to the plant’s network architecture at the same time.

Retention and Readability: The Question That Should Come First

Most historian modernization business cases open with analytics. They should open with retention. The reason is simple: analytics is an ambition that can be deferred, and retention is an obligation that cannot.

What the regulations actually require

Under 21 CFR 211.180(e), production, control, and distribution records specifically associated with a batch must be retained for at least one year after the batch expiration date, and records must remain readily available for inspection throughout the retention period.¹⁷ For a product with a five year shelf life, that is a six year floor from the date of manufacture, and most companies set internal policies well above the floor to cover litigation, product complaints, and post-approval commitments.

Retention alone is not the hard part. Storage is inexpensive. The hard part is readability. The MHRA’s data integrity guidance is explicit that archive arrangements must be designed to permit recovery and readability of the data and metadata throughout the required retention period, and that where a computerized system can no longer be maintained, records should be archived according to a documented strategy before the system is decommissioned. The same guidance notes that for legacy systems, the ability to review data periodically should be verified rather than assumed.

Read that last point carefully. The expectation is not that the site believes old data is readable. It is that the site has checked, on a defined schedule, and can show the check.

Why decade-old time-series data is a specific problem

Historian archives age badly in ways that are easy to miss. Several failure modes recur:

• Archive file format drift. Historian products have gone through major version changes over twenty years. Archives written by an old version are often readable only by a compatibility layer that the vendor supports for a defined window and then retires.
• Tag configuration drift. A tag’s engineering units, scaling, or descriptor may have been changed at some point. The archive stores values. If the metadata that explains what those values mean has been overwritten rather than versioned, old data becomes ambiguous.
• Lost point mapping. Tag names were often generated by the control system integrator using site conventions that changed when the DCS was upgraded. Historical tags may have no surviving map to the equipment they came from.
• Orphaned interfaces. Data collected from a skid that was decommissioned in 2016 is still in the archive, but nothing in the current system describes what the skid was or how its instruments were calibrated.
• Backup media that nobody has restored. Offline archives on tape or on decommissioned storage arrays are frequently untested. The first restore attempt during a migration project is often the first restore attempt ever.

Deciding what has to move and what only has to remain retrievable

Not all historian data carries the same obligation. Treating it as one undifferentiated block is the most common way these projects become unaffordable in effort and schedule. A tiered approach works better.

The tiering exercise is not a technical task. It requires quality, process engineering, and manufacturing to agree, tag by tag for Tier 1 and by tag group below that. It is slow. It is also the single highest-value activity in the whole program, because it is the only thing that turns an unbounded migration into a bounded one.

Compression: The Record You Kept Is Not the Signal You Measured

This is the part of historian modernization that most projects avoid discussing, and it deserves direct treatment because it sits awkwardly against data integrity expectations.

How exception and compression actually work

Process historians were built when storage was expensive and network links to the plant floor were slow. To make continuous collection from thousands of instruments practical, they apply two filters in sequence.

The first is exception reporting, applied at the interface or collector, close to the control system. A new reading is only sent onward if it differs from the last reported value by more than a configured deadband. Readings inside the deadband are treated as noise and dropped at the source. They never reach the archive at all.

The second is compression, applied at the archive. The most widely used method is the swinging door algorithm, which fits a straight line through consecutive readings and stores only the endpoints of each segment, as long as every intermediate reading falls within a tolerance corridor around that line. When a reading falls outside the corridor, the previous value is committed to the archive and a new segment begins. Vendor documentation describes these two mechanisms and their tuning parameters in detail.⁸

The result is that a tag sampled every second may be stored as a few hundred points per day rather than 86,400. On retrieval, the historian interpolates between stored points, so a trend chart looks continuous. Most users have no idea that the smooth line they are reading is largely reconstructed.

Why this matters for data integrity and for analytics

The data integrity concern is not that compression is inherently improper. Reduction of a continuous analog signal to a stored representation is unavoidable in any digital system, and regulators have never expected sites to archive every sample from every instrument at full rate. The concern is that the reduction is an undocumented, uncontrolled transformation applied to GMP records.

If the compression deviation on a bioreactor temperature tag is set wide enough to smooth over a short excursion, that excursion may not appear in the archive at all. An investigator reviewing the trend would see a process that stayed within limits. Nobody falsified anything. The configuration made the excursion invisible. That is a control weakness, and the fact that it originated in an engineering setting rather than a user action does not make it less of one.

The analytics concern is related and better documented. Research on compression in real-time and historical data acquisition systems shows that excessive compression can lose critical patterns, skew statistical measures, and reduce predictive accuracy in downstream models.⁷ A model trained on aggressively compressed data learns the shape of the compression as much as the shape of the process. Multivariate methods are particularly sensitive, because compression is applied per tag, so different tags in the same model are reduced at different rates and their apparent correlations shift.

What to do about it during a modernization

The practical approach is not to eliminate compression. It is to bring it under control and to document it as what it is.

1. Inventory the settings. Extract exception deviation, compression deviation, compression maximum time, and compression minimum time for every tag. This is usually a straightforward export and it is usually the first time anyone has looked at it as a whole.
2. Compare settings against instrument accuracy. A compression deviation set wider than the instrument’s own accuracy is defensible. One set several times wider is a decision that needs a rationale. Many sites find tags where the deviation was set at a vendor default and never revisited.
3. Tighten or disable compression on Tier 1 tags. For the tags tied to critical process parameters, storage is no longer a meaningful constraint. Turning compression off entirely on that subset, or setting it at instrument accuracy, removes the ambiguity for the data that matters most.
4. Treat the settings as configuration under change control. Compression parameters affect the content of a GMP record. They belong in the validated configuration baseline, with changes going through change control and a documented impact assessment, not in an engineer’s discretion.
5. Quantify the effect once, on real data. Take a set of representative batches, collect them uncompressed in parallel for a defined period, and compare the compressed and uncompressed traces. The comparison gives a defensible statement about what compression does to the record at that site, which is far better than an assertion that it does not matter.
6. Decide what the migration preserves. Historical data cannot be uncompressed. What was discarded is gone. The migration should carry across exactly what the archive holds, with the original timestamps and quality flags, and the documentation should be clear that the migrated record reproduces the legacy archive rather than the original instrument signal.

That last point matters for how the migration is described in validation documents. Claiming the new system holds the raw data is inaccurate if the legacy archive never did. Claiming it faithfully reproduces the legacy archive is accurate and provable, and it is what an inspector would actually want to see demonstrated.

Contextualization: A Tag Value Is Not a Batch Record

The second structural problem is that a historian holding raw tags and a historian holding contextualized batch data are different systems with the same name, and the difference determines whether anyone can use the data without a specialist sitting next to them.

What context means concretely

A value of 37.2 with a timestamp of 14 March 2024 at 09:41:16 tells you almost nothing. To use it, you need to know at minimum:

• Equipment identity and hierarchy. Which vessel, in which process cell, in which area, at which site. The ISA-95 equipment hierarchy defines this structure: enterprise, site, area, process cell or line, unit, equipment module.⁹
• Batch identity. Which batch or lot was in the vessel, and which product and strength.
• Procedural state. Which unit procedure, operation, and phase the control recipe was executing. A temperature of 37.2 means something different during inoculation than during a hold or a clean in place cycle.
• Equipment state. Whether the vessel was in production, idle, being cleaned, or in maintenance. Data collected during a CIP cycle mixed into production trending produces nonsense.
• Material genealogy. Which raw material lots, media lots, and buffer preparations were charged.
• Data quality flag. Whether the instrument was reporting good, bad, uncertain, or substituted at that moment, which most historians record and most downstream reports discard.

Without those, every analysis begins with an engineer manually reconstructing what was happening from batch records and shift logs. With them, an analysis that used to take two days takes twenty minutes, and more importantly it can be repeated by somebody else and produce the same answer.

Two historians, same tags, different value

Where the context comes from and who owns it

Context does not originate in the historian. It originates in the systems that run the plant: the batch execution system for procedural state, the MES for batch and material genealogy, the ERP for product master data, the maintenance system for equipment status. ISA-95 describes this layered structure, with Level 3 systems providing the contextualization that Level 4 systems consume.⁹ ISPE’s work on Pharma 4.0 makes the same point from the quality direction, describing automated data gathering and contextualization infrastructure as a precondition for a modernized approach to continued process verification rather than an optional enhancement.²

This has an organizational consequence that most programs discover late. Contextualization is an integration project across three or four system owners, and it usually exposes that the equipment naming in the historian does not match the equipment naming in the MES, which does not match the asset numbering in the maintenance system, which does not match the functional location in the ERP. Reconciling those four hierarchies is often the longest single task in a historian modernization, and it produces no visible output until it is finished.

Two technical patterns dominate current practice. The first uses an asset framework layer inside the historian itself, where equipment templates carry the hierarchy and event frames or capsules mark the batch and phase boundaries. The second publishes contextualized data through a broker layer, using OPC UA information models at the machine level¹³ and a topic structure aligned to the ISA-95 hierarchy for onward distribution.¹⁴ Both work. The choice usually follows the existing control system estate rather than any abstract architectural preference.

The Historian Sits on the IT and OT Fault Line

Every historian modernization runs into the network architecture, and it is worth being concrete about why rather than treating it as a security team objection to be managed.

Why the boundary exists and where the historian falls

Plant networks are segmented for a reason that has nothing to do with data governance. Control systems run equipment that can injure people and destroy product. They are built for availability and determinism, they often run operating systems that cannot be patched on an IT schedule, and they are protected by keeping them separated from general-purpose networks. The Purdue reference model, adopted by the ISA-99 committee as the conceptual basis for the zone and conduit approach in the ISA/IEC 62443 series, describes this as a set of levels: field devices and control at the lower levels, supervisory control above that, site operations including MES and historian servers at Level 3, and enterprise systems at Level 4, with a demilitarized zone between them.¹⁰¹¹

The historian is unusual because its job requires it to touch both sides. It has to read continuously from control systems at Levels 1 and 2, and it has to serve data to engineers, quality reviewers, and reporting systems at Level 4. NIST’s guide to operational technology security places historian servers at the operations level alongside MES, and treats the boundary between operations and enterprise as a primary control point.¹⁸ CISA’s segmentation guidance makes the same architectural point for industrial environments generally.¹²

Poorly configured historians become the path an attacker uses to move from the business network into the control network. The safe pattern is well established: the collection function pulls from below and never accepts inbound connections from above, and anything that serves data upward does so from a separate instance in the DMZ.

What moving shop floor data to an enterprise or cloud platform actually changes

The proposal in most modernization programs is to move historian data into an enterprise data platform, frequently a cloud one, so that analytics can run across sites. That is a reasonable ambition. The consequences need to be stated accurately.

Why a read-only replica is often the right compromise

The pattern that resolves most of this tension is a one-way replica. The plant historian stays where it is, in the operations zone, collecting from control systems and serving the site. A replication process pushes data outward into a read-only instance in the DMZ or into the enterprise platform. Nothing writes back. The plant retains the record. The enterprise gets the analytics surface.

This is not architectural timidity. It solves several problems at once:

• Security direction is fixed. Data flows one way, out. There is no inbound path to the control network to review, monitor, and defend.
• Validation scope stays bounded. The site historian remains the GMP record under its existing qualification. The replica is a derived copy, and its use determines whether it needs its own controls rather than dragging the whole enterprise platform into GMP scope by default.
• The plant is insulated from enterprise change. Cloud platform upgrades, schema changes, and analytics experiments cannot affect data collection or the site’s ability to review batches.
• Migration risk is contained. A replica can be built and proven while the legacy system continues to run untouched, which is the safest possible starting position for any historian project.

The trade-off is real: a replica means duplicated storage, a reconciliation process to prove the copy is complete, and a standing requirement to keep the two in agreement. That reconciliation is not optional and it is not a one-time test. It is a scheduled control with a defined tolerance and an owner.

The Analytics Ambition, in the Honest Order

Almost every historian modernization is funded on an analytics case. Predictive maintenance, golden batch analysis, multivariate process monitoring, yield optimization, early deviation detection. Those are genuine opportunities. The sequencing in most business cases is backwards, and the consequence is predictable.

Why the business case is usually written in the wrong order

The typical proposal reads: replace the historian, move the data to a modern platform, apply machine learning, realize benefits. The implied assumption is that the constraint is the storage technology and the analytics tooling. In most plants, it is not. The constraint is that the data has no context, so every analysis starts with weeks of manual reconstruction that is never documented and never reusable.

ISPE’s work on data science for pharmaceutical development and production makes this point repeatedly across both parts of its treatment: the difficulty is in aligning time-value pairs with discrete quality measurements and in establishing context across data sources, not in the modeling technique.³⁴ Reviews of machine learning in upstream bioprocessing describe the same constraints from the technical side, noting limited usable historical data, heterogeneous process conditions, and infrequent feedback as the primary obstacles rather than algorithm selection.¹⁶

What a model trained on uncontextualized tags does in a real investigation

This is the specific failure worth understanding, because it is where the reputational damage happens.

A team builds a model on three years of historian data to flag batches at risk of a low yield. It performs well on held-out data. It goes into use as an early warning. Six months later it flags a batch, the site investigates, and quality asks the obvious questions: which parameters drove the flag, what data was the model trained on, how was that data selected, and can you show that the training data excluded periods when the equipment was being cleaned or when a sensor was reading bad quality.

If the training set was pulled as raw tag ranges without equipment state or phase framing, the honest answers are that some CIP cycles were probably included, that bad-quality readings were probably treated as real values, and that batch boundaries were approximated from a tag threshold rather than from the batch record. None of that is fraud. All of it means the model cannot be defended in the investigation, and the flag becomes an unexplained finding that has to be closed out rather than a piece of evidence.

The second-order effect is worse than the first. Once a model has failed in an investigation, quality’s position on the next one hardens, and reasonably so. Sites that push analytics ahead of contextualization frequently set their own program back by two years.

The sequence that holds

Steps one through three deliver measurable benefit on their own. That matters for funding, because it means the program does not have to hold its value promise until the final step to show a result.

A Migration Approach That Survives an Inspection

With the preceding sections settled, the migration itself becomes a manageable engineering exercise. The approach below assumes a Tier 1 dataset that carries a regulated retention obligation.

Decide the target pattern before anything else

There are three viable end states and they carry very different effort and risk profiles.

The forward cutover is underused and often the right answer. It removes the largest single block of migration effort, it lets the new system prove itself on live data before anything historical depends on it, and it converts the legacy system from an operational dependency into an archive with a much smaller support footprint. Its weakness is that investigations spanning the cutover date need two systems, which is manageable with a documented procedure and becomes irrelevant once the cutover date passes out of the retention window.

The parallel run and what it has to prove

Whichever pattern is chosen, both systems should collect the same data at the same time for a defined period before anything is switched off. Parallel running is the standard approach for high-risk regulated cutovers because it produces direct evidence rather than inference.

The parallel period should be long enough to cover a representative set of production events, not a fixed number of weeks chosen for the schedule. Concretely, it should include at least: several complete batches of each product family running on the affected equipment, at least one full clean in process and clean in place cycle, a planned equipment shutdown and restart, a network interruption of the collection path, and a historian server restart. Those are the conditions under which collection systems differ from each other, and the parallel run exists to find those differences.

Two points about the value-for-value comparison deserve emphasis. First, the tolerance must be defined and justified before the test runs, not selected afterward to fit the result. Second, exact equality should not be expected if the two systems apply different compression. Setting an acceptance criterion of exact match on a system with different default compression behavior guarantees a failed test and a scramble to explain it. State the expected difference in the protocol, and test against that.

What Has to Be True Before You Retire the Legacy System

The retirement decision is where historian projects most often stall, because it is the point at which the site has to say, in writing, that it no longer needs the system holding its process history. That statement needs support.

The retirement gate list

Before the legacy historian is decommissioned, the following should be complete and approved:

1. The parallel run is closed with a signed report. All verification tests executed, all deviations closed, all differences explained. Not summarized in a slide. A report that the quality unit has approved.
2. Every retained record has a proven home. For each data tier, a documented statement of where the data now lives, how it is retrieved, and by whom. Tier 4 data has a disposal record approved under the records retention procedure.
3. Retrieval has been demonstrated by someone who was not on the project. A quality reviewer or process engineer independently retrieves a defined set of historical records from the new system using the documented procedure, without project support.
4. Downstream systems and reports are cut over and verified. Every CPV chart, product quality review extract, dashboard, and scheduled report that read from the legacy system now reads from the new one and produces the same result.
5. Procedures are updated and people are trained. Investigation procedures, CPV procedures, and data review SOPs reference the new system. Training is complete and recorded for everyone who retrieves process data.
6. Periodic readability verification is scheduled. A recurring check confirming that the oldest retained data remains retrievable and readable, with defined frequency, method, and owner. This is a standing control, not a project deliverable.
7. The decommissioning itself is documented. A decommissioning record covering final backup, verification of that backup, data disposition, hardware disposal, and the date after which the legacy system is no longer available. Retain it for the same period as the records the system held.
8. A defined fallback exists until the gate is fully closed. Keep the legacy system powered, isolated, and read-only for a stated period after cutover. Turning it off on the day the new system goes live removes the only real fallback the site has.

When migration is not viable

Sometimes the legacy archive genuinely cannot be migrated. The format is proprietary and the vendor no longer supports extraction. The volume makes verification impractical within any reasonable schedule. The tag metadata needed to interpret the values no longer exists.

In that situation the archive-and-retain path is legitimate, and it is better to take it deliberately than to discover it halfway through a failing migration. Freeze the legacy system, restrict it to read-only access, document a retrieval procedure that a trained reviewer can follow without vendor support, verify retrieval on a schedule, and record the whole arrangement as a controlled archive with an owner and an end date tied to the retention obligation. The regulatory expectation is readability and retrievability throughout the retention period, not residence in the current platform.

What is not acceptable is the silent version of this: leaving an unsupported server running in a corner with no documented procedure, no verified retrieval, and no owner, on the assumption that it will still work if anyone ever asks. That is the arrangement inspectors find, and the finding is not about the technology.

Governance after go-live

A modernized historian degrades in the same way the old one did unless a few controls are put in place at the start.

Conclusion

Historian modernization is presented as an infrastructure decision and it is really a records decision. The system holds a regulated record with a long retention obligation, that record was transformed on the way in by settings most sites have never reviewed, and it is close to unusable without context that lives somewhere else. Any program that treats the work as a platform swap will find these problems late, in validation, when they are most expensive to resolve. Any program that starts with retention, compression, and context will find that the platform question becomes much easier and sometimes turns out to be less urgent than assumed.

The honest sequencing is the whole message here. Contextualize before you migrate, because the context model is portable and the tag data is not. Understand and control your compression before you promise anyone a faithful record. Establish which system is the record before you argue about cloud architecture. Prove the new system on live data through a parallel run that covers real production conditions before you commit historical data to it. And keep the legacy system available, read-only, until an independent reviewer has retrieved what they need from its replacement. None of that is exotic. It is simply the order that produces defensible outcomes rather than a system that works until the first time it is questioned.

Sakara Digital works with pharma and biotech organizations facing exactly this kind of decision, where a long-lived plant system carries more regulatory weight than its place on the architecture diagram suggests. If you are weighing a historian modernization and want an independent read on whether replacement, forward cutover, or contextualization on the existing platform is the right first move, we are happy to have that conversation.

For Further Reading