In This Article
- Executive Summary
- Why a Validated SaaS Outage Is a Quality Event
- What Regulators Expect When a GxP System Goes Down
- Minutes 0 to 15: Confirm, Declare, and Assign
- Minutes 15 to 30: Choose the Fallback
- Minutes 30 to 60: Recording GxP Data During Downtime
- Working With the Vendor During the Outage
- After Recovery: Reconciling Data and Restoring the Validated State
- Building the Playbook Before You Need It
- Conclusion
- For Further Reading
- References & Sources
Executive Summary
Most pharma and biotech companies now run core GxP work on software they do not host: quality management, document control, LIMS, EDC, safety, and regulatory information systems. When one of those systems goes down, the vendor controls the fix, but the regulated company still owns the records, the decisions, and the compliance risk. A validated SaaS outage is a quality event first and an IT ticket second.
This article gives a first-hour playbook built on what EU GMP Annex 11, ICH E6(R3), MHRA, PIC/S, and FDA already expect. It covers who declares downtime and who decides whether GxP work continues, how to choose between pausing, controlled paper, or an alternate system, how to record data so it stays attributable and contemporaneous, and how to run the conversation with the vendor when the facts are still unclear.
The second half covers what happens after the system returns: a controlled cutover back to electronic records, back-entry with a second-person check, reconciliation of interfaces and any data the vendor lost, and a documented decision that the system is back in its validated state. We close with how to build and drill the playbook so the first hour of a real outage is not the first time anyone reads it.
Why a Validated SaaS Outage Is a Quality Event
When an on-premises system failed ten years ago, the people who could fix it usually worked down the hall. The quality team could walk over, ask what broke, and get a straight answer about whether data was at risk. With software as a service (SaaS), that conversation now runs through a vendor support portal, a status page, and a support engineer in another time zone who may know less about your configuration than your own system administrator does.
The regulatory position has not moved with the hosting model. The regulated company is still responsible for the integrity of its GxP records, for the continuity of the processes those records support, and for showing an inspector what happened. Handing the infrastructure to a vendor moves the work of keeping the system running. It does not move the accountability for what your people do when it stops.
Outages Happen Even When Your Vendor Does Nothing Wrong
A SaaS application depends on layers you never see: a cloud provider, a content delivery network, an identity provider, and the endpoint software on your own laptops. Any one of them can take a validated system out of reach for your users, and several large recent events show how.
AWS traced its October 2025 event to a latent defect in the automated system that manages DNS records for its DynamoDB database service. Customers and other AWS services that depended on DynamoDB could not open new connections, and the effects spread to load balancers and new server launches over the following hours.1 Any SaaS product built on that region could have gone down without a single change to its own code.
In July 2024, CrowdStrike released a faulty content update to its Windows sensor at 04:09 UTC and reverted it at 05:27 UTC, 78 minutes later.4 Machines that had already received it crashed, and Microsoft posted manual remediation instructions and scripts to help customers recover.2 For a GxP user, the SaaS system itself may have been fine. The workstation used to reach it was not. From the lab bench, that is still an outage.
Cloudflare’s November 18, 2025 incident began at 11:20 UTC. Core traffic was largely flowing again by 14:30, and all systems were normal by 17:06.5 Two details from Cloudflare’s own write-up matter for the first hour of any outage. The team first suspected a large attack, which turned out to be wrong. And Cloudflare’s status page, hosted entirely outside Cloudflare, went down at the same time by coincidence, which pushed some of the team further toward the wrong explanation.5 If the provider’s own engineers can misread the first signals, your users and your vendor’s first support reply can too.
What Changes When the System Is Validated
For an ordinary business application, an outage is a productivity problem. For a validated GxP system, it raises questions the quality unit must answer:
- Can the GxP activity continue at all, or must it stop until the system returns?
- If it continues, where is the official record created, and who controls that record?
- Were any records lost, partly written, or duplicated when the system failed?
- Did the vendor’s recovery change the system in a way that affects its validated state?
- Do any regulatory clocks, such as safety reporting or batch disposition timelines, keep running while the system is down?
None of those questions can be answered well by someone improvising at 7:15 on a Monday morning. They need to be answered in advance, written into a procedure, and practiced. That is the point of a first-hour playbook.
The Scale of the Problem When Workarounds Fail
Health care gives the clearest public evidence of what happens when a critical third-party service stays down for a long time. After the February 2024 cyberattack on Change Healthcare, an American Hospital Association survey of about 1,000 hospitals found that 94% were experiencing a financial impact, and 74% reported impacts to direct patient care. Most hospitals said their workarounds were very labor intensive and costly.6 Pharma operations are different, but the lesson carries over: a workaround that nobody designed in advance is slow, error-prone, and hard to unwind afterward.
What Regulators Expect When a GxP System Goes Down
There is no single regulation titled “SaaS outage.” The expectations are spread across computerized system, documentation, data integrity, and clinical guidance. Read together, they are consistent and fairly specific.
EU GMP Annex 11: Business Continuity and Incident Management
The current EU GMP Annex 11, in force since 2011, has a dedicated business continuity clause. Section 16 says that for computerized systems supporting critical processes, “provisions should be made to ensure continuity of support for those processes in the event of a system breakdown (e.g. a manual or alternative system).” It adds that the time needed to bring the alternative arrangements into use “should be based on risk” and appropriate for the system and business process, and that these arrangements “should be adequately documented and tested.”7
Three other Annex 11 clauses shape what happens during and after the outage. Section 13 says all incidents, not only system failures and data errors, should be reported and assessed, and that the root cause of a critical incident should form the basis of corrective and preventive actions. Section 6 says that for critical data entered manually, there should be an additional check on the accuracy of the data, done by a second operator or by validated electronic means. And Section 3.1 requires formal agreements with third parties, including service providers, with clear statements of their responsibilities.7 Section 6 matters most after recovery, when paper records are keyed back into the system.
A revised Annex 11 was published as a consultation draft in 2025. As of late September 2026, the European Commission’s EudraLex Volume 4 page still lists the January 2011 revision as the current Annex 11.8 The draft does not carry a standalone business continuity chapter. Instead, it asks for a tested disaster recovery plan that, where applicable, ensures continuity of operation within a defined Recovery Time Objective (RTO), and it spells out what contracts with service providers should cover, including incident reporting, SLAs and KPIs, answer times, and resolution times.9 Plan against the 2011 text, which is the one in force, and use the draft as a signal of where inspector expectations are heading.
MHRA and PIC/S: Put Continuity in the Contract
The MHRA data integrity guidance speaks directly to cloud and outsourced services. It says the responsibilities of the contract giver and acceptor should be defined in a technical agreement or contract, and states: “Business continuity arrangements should be included in the contract, and tested.” It also says that “Appropriate arrangements must exist for the restoration of the software/system as per its original validated state, including validation and change control information to permit this restoration.”10 That second sentence is the regulatory basis for the “restore the validated state” step later in this article.
PIC/S guidance PI 041-1 asks companies to give consideration to systems with a higher potential for disaster, malfunction, or situations in which the system becomes inoperative, when deciding the extent of controls.11 Its section on paper documents is just as important for downtime, because downtime paper is still a GMP record. We return to it below.
ICH E6(R3): Contingency Procedures for Clinical Systems
For clinical trial systems, ICH E6(R3) section 4.3.6, titled System Failure, says contingency procedures “should be in place to prevent loss or lack of accessibility to data essential to participant safety, trial decisions or trial outcomes.” The sponsor responsibilities in section 3.16.1 also list backup and disaster recovery among the computerized system requirements a sponsor must make sure are addressed.12 E6(R3) names randomization, dosing, and collection of endpoint data as examples of critical functionality, which gives a useful short list of trial functions whose downtime procedure must be written and tested.12
FDA and EU Documentation Rules Apply to Downtime Paper
FDA’s data integrity guidance is blunt about informal notes: “It is not acceptable to record data on pieces of paper that will be discarded after the data are transcribed to a permanent laboratory notebook.”13 EU GMP Chapter 4 says records should be made or completed at the time each action is taken, and that “It should be clearly defined which record is related to each manufacturing activity and where this record is located.”14 During an outage, those two rules together mean the downtime form is the original record, it must be controlled, and it must be kept after its contents are keyed into the system.
| Source | What It Asks For | Where It Applies in the Outage |
|---|---|---|
| EU GMP Annex 11, Section 16 (2011) | Documented, tested manual or alternative arrangements, with activation time based on risk | Choosing and activating the fallback |
| EU GMP Annex 11, Sections 6 and 13 | Second check of manually entered critical data; incidents reported, assessed, and root-caused | Back-entry and the incident record |
| Draft Annex 11 (2025 consultation) | Tested disaster recovery plan with a defined RTO; contracts covering incident reporting, answer and resolution times | Vendor agreement and communication |
| MHRA data integrity guidance (2018) | Business continuity in the contract and tested; restoration to the original validated state | Vendor agreement; restoring the validated state |
| PIC/S PI 041-1 | Controlled issue and reconciliation of blank forms | Downtime paper forms |
| ICH E6(R3), 4.3.6 and 3.16.1 | Contingency procedures for data essential to participant safety and trial outcomes | Clinical systems (EDC, IRT, eCOA, safety) |
| FDA data integrity Q&A (2018); EU GMP Chapter 4 | No discarded transcription notes; records made at the time of the action and located in a defined place | Recording data during downtime |
The practical reading: regulators do not expect a SaaS system never to fail. They expect you to have decided in advance what happens when it does, to have tested that decision, and to be able to show that every GxP record created during the outage is attributable, contemporaneous, complete, and traceable into the system afterward.
Minutes 0 to 15: Confirm, Declare, and Assign
The first fifteen minutes decide whether the next several hours are controlled or chaotic. The goal is not to fix anything. It is to establish that there is an outage, name the people in charge, and stop anyone from creating records in places nobody will find later.
Confirm the Scope Quickly, Without Waiting for the Vendor
The first report usually comes from a user: a login that hangs, a save that fails, a page that times out. The person receiving it, often the service desk or the system administrator, should run a short, pre-written check:
- Is it one user, one site, or everyone? Try from a second machine and a second network.
- Is it the whole application or one module, such as reporting or e-signature?
- Is the vendor’s status page showing an incident? Treat it as one input, not proof either way.
- Are other SaaS tools that share the same identity provider also failing? If so, the problem may be sign-in, not the application.
- Is the system slow rather than down? A system that takes three minutes to save a record can be as disruptive as one that is offline.
The last point is easy to skip. The ONC SAFER guide on contingency planning, written for electronic health records, explicitly calls for functional system downtimes, meaning unacceptably slow response time, to be identified and addressed proactively.15 Your procedure should define what “unacceptably slow” means for each critical system, so nobody has to argue about it in the moment.
Declare Downtime Against Written Criteria
NIST’s incident response guidance, SP 800-61 Revision 3, recommends that incidents be declared when adverse events meet defined incident criteria.16 The same principle works for GxP downtime. Write the criteria down in advance: for example, a critical system unavailable or unusably slow for a set number of minutes during working hours, or any sign that data may have been lost or corrupted. When the criteria are met, the downtime lead declares downtime. They do not wait for the vendor to confirm, because the vendor may take an hour to acknowledge the incident, and the GxP work in front of your people cannot wait that long without a decision.
Declaring downtime and later finding out it was a short blip is a minor inconvenience. Not declaring it, and letting staff improvise for two hours, creates records you will spend weeks reconstructing.
Who Decides: Four Named Roles
Annex 11 already expects close cooperation among the process owner, the system owner, qualified persons, and IT, with defined responsibilities.7 For downtime, turn that into four named roles, each with a named backup:
Downtime Lead (Business Process Owner)
Declares downtime, decides whether the business process pauses or moves to the fallback, and declares the return to normal. Owns the business decision, not the technical fix.
Quality Decision-Maker
Agrees that GxP work may continue under the fallback, sets any limits (for example, no batch disposition until the system returns), and opens the deviation or incident record.
System Owner or IT Liaison
The single point of contact with the vendor. Opens the ticket, logs every exchange with times, and translates vendor updates into plain facts for the downtime lead.
Communications Lead
Tells affected users what is happening, what to do, and when the next update will come. Uses channels that do not depend on the failed system or its sign-in.
A common failure: the system owner is also the person expected to decide whether manufacturing or lab work continues. In the first hour, that person is busy with the vendor. Separate the technical liaison from the business decision, and make sure the quality decision-maker is a named person with authority, not “QA” as a group.
Freeze Uncontrolled Record-Keeping
The last step in the first fifteen minutes is a short instruction to all affected users: stop entering data into the system, and do not record GxP data on sticky notes, in personal spreadsheets, in email, or in chat messages. Tell them where the downtime kit is and who will tell them what to do next. This instruction prevents a common downtime problem: data that exists only in an informal place and cannot be shown to be complete.
Minutes 15 to 30: Choose the Fallback
Annex 11 describes the alternatives as “a manual or alternative system.”7 In practice there are four options, and a mature playbook has already chosen among them for each critical process, so the downtime lead is confirming a plan rather than designing one.
Option 1: Pause the Activity
Many GxP activities can wait. Approving a change control, closing a CAPA, releasing a controlled document, or signing off a validation report rarely has to happen in the next four hours. For these, the safest fallback is to stop and resume in the system when it returns. Pausing creates no hybrid records and no back-entry, and nothing needs to be reconciled. The playbook should say which activities are “pause by default,” so nobody feels pressure to invent a paper approval process in the moment.
Option 2: Controlled Paper
Where the work cannot stop, such as an in-process manufacturing step, a time-sensitive lab test, or a clinical site visit, controlled paper forms are the usual fallback. “Controlled” is the key word. These are pre-approved downtime forms that capture the same critical data as the system, issued and reconciled under the same rules as any other GMP form. The next section covers how to design and use them.
Option 3: An Alternate System or Read-Only Copy
Some companies keep a validated alternate: a local data store on an instrument, a secondary system that can take the work temporarily, or a read-only copy of key data. The SAFER guide recommends a read-only backup for electronic health records that is tested regularly and allows users to print, so clinicians can see current information even when the main system is down.15 In pharma, the equivalent is often a regularly refreshed, controlled export of the information people need to keep working safely: current specifications, open batch status, active study subject lists, or current effective SOPs.
Option 4: Continue in the Parts That Still Work
Partial outages are common. The review module may be down while data capture works, or the web client may fail while a mobile app still connects. Continuing in the working parts can be sensible, but only if the playbook has already confirmed that the working module’s records are complete on their own and that no required control, such as an audit trail or e-signature, depends on the failed component.
| System Type | Typical First-Hour Fallback | What to Watch |
|---|---|---|
| eQMS and document management | Pause approvals; use controlled, time-stamped printed copies of current SOPs | Operators need current procedures; printed reference copies must show when they were printed and how long they are valid |
| LIMS | Controlled paper worksheets; instrument data held in the instrument’s own storage | Sample identity, chain of custody, and instrument data that must later be linked to the right sample |
| MES or electronic batch record | Pre-approved paper batch record sections for the affected steps | Critical process parameters, second-person verification, and a clear point where the paper takes over and hands back |
| EDC, eCOA, or IRT (clinical) | Site source documents; emergency randomization or dosing procedure agreed in advance | Randomization, dosing, and endpoint data are named as critical functionality in ICH E6(R3) |
| Safety database | Controlled case intake log | Reporting clocks continue; case receipt dates must be captured accurately |
How Fast the Fallback Must Start
Annex 11 leaves the activation time to risk: it must be appropriate for the system and the business process it supports.7 Health care offers useful reference points. The SAFER guide recommends paper forms on hand for at least 8 hours of care in each area, and a policy that says when a warm-site backup should be activated, ideally before an unplanned outage reaches 2 hours.15 A pharma company will set its own numbers, but it should set them. “We will switch to paper after 30 minutes of downtime on the packaging line, and after four hours for the document system” is a decision. “We will assess the situation” is not.
Minutes 30 to 60: Recording GxP Data During Downtime
By the half-hour mark, users should be working from the fallback. The quality of the records they create now decides how hard the recovery will be. The evidence from health care is sobering. In a study of downtime at a 300-bed hospital, researchers found laboratory results were delayed by an average of 62% during downtime, and the archived data were incomplete due to inconsistencies in the downtime paper records. Staff interviews showed the delays were often due to improper procedural execution and incomplete or incorrect documentation.17 Paper under pressure tends to be incomplete unless the form and the process make completeness easy.
Design Downtime Forms That Mirror the System
A good downtime form asks for exactly what the system would have captured, in the same order, with the same units and the same required fields. Each form should include:
- A unique form number, issued from a register, so the set can be reconciled later.
- The downtime event reference, so every form links to one incident record.
- Fields for the actual date and time of each observation or action, taken from a defined reference clock.
- The identity of the person doing the work, and of any second person verifying it.
- Fields for the identifiers the system would normally generate or check, such as sample IDs, lot numbers, or subject numbers.
- A block for the back-entry step: who entered it, when, and who verified the entry.
PIC/S PI 041-1 sets out the controls for blank forms, and they apply fully to downtime forms: record who issued copies and when, allocate a unique identifier to each blank document and record its issue in a register, and reconcile critical GMP blank forms after use to ensure the records are accurate and complete.11 The downtime kit should hold pre-numbered forms and the register, stored where people can reach them without the failed system.
Keep the Paper as the Original Record
When a user writes a result on a controlled downtime form, that form is the original record. EU GMP Chapter 4 expects records to be made at the time each action is taken.14 FDA says it is not acceptable to record data on paper that will be discarded after transcription.13 So the paper is retained, linked to the electronic entry, and treated as part of a hybrid record for the life of the data. Nobody should shred downtime forms once “everything is in the system.”
Signatures, Approvals, and Printed SOPs
If an approval must happen during downtime and cannot wait, it is a handwritten signature on the controlled form, dated with the actual date. When the system returns, do not re-create that approval as an electronic signature dated as if it happened during the outage. Record in the system that the approval was given on paper, reference the form, and attach a scan if your procedure allows. The system’s audit trail will show the true entry time, and your records should agree with it.
If the document management system is down, operators may lose access to current SOPs. PI 041-1 allows printed reference copies of documents that are not records without reconciliation, provided they are time-stamped when generated and their short-term validity is marked on the document.11 A controlled, regularly refreshed set of current SOPs for critical operations, with a printed-on date and a “valid until” date, is one of the most useful items in any downtime kit.
What inspectors look for after an outage: GxP data that appears in the system with no matching downtime form; forms that are missing from the register; back-entered records dated to the original activity time with no note explaining the late entry; results transcribed without a second-person check; and approvals that seem to have been given electronically while the system was down.
Keep a Running Log
The downtime lead should keep a timed log from the moment of declaration: when the fallback started in each area, which forms were issued, what the vendor said and when, and every decision taken. This log becomes the main source for the incident record. It also answers the question an inspector may ask months later: what happened that day, in what order, and who decided what.
Working With the Vendor During the Outage
The vendor controls the repair. Your job is to get the facts you need to make quality decisions and to hold the vendor to the commitments in your agreement.
One Voice, One Ticket, One Log
Only the named system owner or IT liaison should talk to the vendor about the incident. When five users open five tickets, the vendor’s support team spends time merging them, and your company receives five slightly different answers. The liaison opens one priority ticket, gives the incident reference, and logs every exchange with the time it happened. NIST recommends following the rules and protocols defined in contracts for incident information sharing between the organization and its suppliers.16 That only works if your contract has such rules, which is a job for procurement and quality long before any outage.
What to Ask in the First Call
Vendors’ first responses tend to be general: “We are aware of an issue and are investigating.” Push, politely, for the answers that affect your quality decisions:
- Scope. Is the incident affecting all customers, one region, or only your tenant? Is it the application, the sign-in service, or an underlying cloud provider?
- Data at risk. Were any transactions in progress lost or partly written? If the vendor restores or fails over, what is the recovery point, meaning the last moment from which data is guaranteed?
- Recovery approach. Will the vendor fail over to a secondary site, roll back a recent change, or apply a fix? Each of these has different implications for your validated state.
- Next update. When will the next update come, even if there is no news? A fixed update time lets your communications lead set expectations.
- Root cause report. Will there be a written post-incident report, and when? You will need it for your incident record.
Treat Early Information as Provisional
The Cloudflare example shows why. Its engineers first suspected an attack, and a coincidental failure of the separately hosted status page reinforced that wrong theory.5 Early vendor updates may be similarly wrong. Record them, act on what is certain, and do not make an irreversible quality decision, such as rejecting a batch or discarding a sample, based on a vendor’s first guess about the cause.
What the Contract Should Already Say
The MHRA expects business continuity arrangements in the contract, and tested.10 The draft Annex 11 lists what a service contract should agree on, including incident reporting and oversight, SLAs and KPIs, answer times, and resolution times, as well as support during regulatory inspections.9 When you negotiate or renew a GxP SaaS agreement, check that it covers:
- How and how fast the vendor notifies you of an incident affecting your tenant, rather than relying on you to notice.
- The vendor’s recovery time and recovery point commitments, and how they are tested.
- Whether a failover or emergency fix is performed under the vendor’s change control, and what evidence you receive.
- A commitment to provide a written root cause report within a set time.
- Access to your data, including audit trails, during and after an incident.
An SLA credit for missed availability is useful to procurement, but it does nothing for your records. The terms that matter for quality are notification, evidence, and data access.
After Recovery: Reconciling Data and Restoring the Validated State
When the vendor reports that service is restored, the temptation is to tell everyone to log back in and carry on. Resist it. The return to normal needs as much control as the switch to downtime, because this is where most data integrity problems are created.
Verify Before You Resume
NIST recommends verifying the correctness and adequacy of restoration actions before putting a restored system online, and working with system owners to confirm successful restoration and the return to normal operations.16 For a validated SaaS system, a short, pre-written post-recovery check might cover:
- Users can sign in with their normal accounts and roles, and access rights are as they were.
- The audit trail is on and recording, with correct time stamps and time zone.
- Electronic signatures work and are linked to their records.
- The last records entered before the outage are present and complete.
- Interfaces to instruments, ERP, or other systems are running, and any queued messages are accounted for.
- Key configuration settings, such as workflows, required fields, and calculation settings, match the approved configuration.
This is a targeted check, not a revalidation. Its purpose is to confirm that the system you are going back into is the system you validated.
Declare a Single Cutover Point
The Lyon and colleagues account of a health district’s downtime and recovery procedure describes two useful rules. The recovery phase starts only when the system is formally announced ready, and staff keep prescribing and recording medication administration on paper “until reconciliation has been completed.”18 The same logic applies to GxP work. The downtime lead announces one cutover time. Before it, the paper is the record. After it, the system is. Any activity in progress at the cutover finishes on paper and is back-entered with the rest.
Back-Enter With a Second-Person Check
Back-entry is transcription of critical data, so Annex 11 Section 6 applies: an additional accuracy check by a second person or validated electronic means.7 Set up the back-entry so that:
- Each entry references its downtime form number.
- The actual activity date and time are recorded in the proper fields, and the system’s own entry time stamp is left as it is. The two times will differ, and they should.
- A comment or reason code marks the record as a late entry from downtime, where the system allows it.
- A second person verifies each critical entry against the form and records the verification.
- Scans of the forms are attached or linked, if your procedure uses scanning.
Plan the effort honestly. The same health district found it needed 2 to 3 hours for a doctor and pharmacist pair to reconcile medication charts in a high-use area, and 30 to 45 minutes in lower-use areas. It finished recovery within 6 hours after an 8-hour planned downtime.18 Those numbers will not transfer directly to a QC lab or a packaging line, but the lesson does: reconciliation takes real time and trained people, and the playbook should estimate it for each area.
Reconcile the Whole Set, Not Just the Forms You Have
Reconciliation should answer four questions, and the answers belong in the incident record:
Are All Issued Forms Accounted For?
Compare the form register with the forms returned. Every issued form is either completed and entered, voided with a reason, or returned unused.
Is Every Completed Form in the System?
Check that each completed form has a matching back-entered record, verified by a second person.
Did Anything Happen Outside the Forms?
Look for activity that should have produced a form but did not: instrument runs with no worksheet, samples received with no log entry, users who kept working in a module that still responded.
What Did the System Lose or Duplicate?
Check records in progress at the moment of failure, interface messages that were queued and replayed, and any gap between the vendor’s recovery point and the outage.
The fourth question needs particular care. If the vendor recovered from a backup or failed over to a secondary site, anything written between the recovery point and the failure may be gone. Those records were never on paper, because nobody knew the system had failed yet. Find them by comparing the last records users remember entering with what the system now shows, and by checking instrument logs and interface message queues. Re-enter lost records from their original sources, with a note explaining why. Duplicates from replayed interface messages need the opposite treatment: identify them, then correct them under your normal data correction procedure with a documented reason.
Decide Whether the Validated State Is Intact
The MHRA expects arrangements to restore the system to its original validated state, including the validation and change control information needed to do so.10 After a SaaS outage, the quality decision rests on the answer to one question: did the recovery change the system?
- No change to your system. The cause was upstream, such as a network or cloud provider failure, and the vendor restored service without changing the application or your configuration. The post-recovery check is usually enough, documented in the incident record.
- A vendor fix or rollback. The vendor deployed a hotfix or reverted a release. Treat it as an unplanned change. Get the vendor’s description of what changed, assess its impact against your validated functions, and run targeted verification of the affected functions under your change control.
- Failover or restore. Your tenant now runs from a secondary site or from restored data. Confirm with the vendor that the failover was performed under their qualified disaster recovery process, confirm your configuration and audit trails came across intact, and reconcile any data lost between the recovery point and the failure.
Exit criteria for closing the outage: post-recovery check completed and signed; cutover time recorded; all downtime forms reconciled against the register; all back-entries verified by a second person; lost and duplicate records identified and corrected; vendor root cause report received and assessed; validated state decision documented by quality; and CAPA opened where the root cause or your own response showed a gap.
Close the Incident Properly
Annex 11 Section 13 asks for every incident to be reported and assessed, and for the root cause of a critical incident to form the basis of CAPA.7 For a SaaS outage, the root cause of the system failure usually belongs to the vendor, and your record should include their report. But your own response has its own root causes. Did it take 90 minutes to declare downtime because nobody knew the criteria? Were forms missing from the kit? Did the call tree depend on the same sign-in service that failed? Those are your gaps to fix. NIST recommends closing recovery with an after-action report covering the incident, the response and recovery actions, and lessons learned.16 The SAFER guide goes further for long events, recommending an in-depth review of unexpected downtimes lasting over 24 hours using root cause or failure modes and effects analysis.15
Building the Playbook Before You Need It
Everything in the first hour depends on work done months earlier. A playbook that exists only as a paragraph in a validation plan will not help at 7:15 on a Monday. Here is what a working one contains.
Start From a Tiered System List
List every GxP SaaS system and rank it by how long each supported process can tolerate the system being unavailable. That tolerance, not the vendor’s SLA, sets your activation time. A packaging line that needs an electronic batch record may tolerate 30 minutes. A training records system may tolerate several days. Record the answer for each system, with the fallback option chosen for each critical process. The SAFER guide also recommends keeping a list of system-to-system interfaces, reviewed regularly, with a note of any that carry legal or regulatory notification duties if they go down.15 Clinical and safety systems are the obvious candidates for that note in pharma.
Assemble the Downtime Kit
For each critical process, the kit holds:
- Pre-approved, pre-numbered downtime forms and the issue register.
- A one-page job aid: how to declare, who to call, what not to do.
- Current printed reference copies of the SOPs needed to keep working, with printed-on and valid-until dates.
- The contact list and call tree, on paper.
- The downtime and recovery procedure itself, on paper.
The SAFER guide recommends a paper copy of the downtime and recovery policy in each clinical unit and another off site, with the policy reviewed at least every 2 years.15 The reasoning holds for a GMP site. A procedure stored only in the system that just failed is no procedure at all.
Make Communication Independent of the Failed System
The SAFER guide recommends that downtime communication not rely on the same computing infrastructure as the failed system, and names email, websites, and voice-over-IP as examples of channels that may fail alongside it. It suggests a mobile phone-based call tree instead.15 Many pharma companies now run email, chat, and their GxP SaaS systems behind one identity provider. If sign-in fails, all of them fail together. Test whether your downtime communications still work when single sign-on does not.
Drill It, Including the Recovery
Annex 11 requires the alternative arrangements to be documented and tested.7 The SAFER guide recommends unannounced downtime drills at least once a year, and training that covers both downtime and recovery procedures.15 A useful GxP drill has three parts:
Tabletop Walk-Through
The four named roles talk through a realistic scenario, such as a vendor-side outage during a batch or a sign-in failure during a site monitoring visit, using the written procedure. Gaps in decision rights show up quickly.
Hands-On Downtime Exercise
Users in one area complete a set of realistic entries on downtime forms, drawn from the kit, under time pressure. Check the forms for completeness afterward.
Recovery Exercise
Back-enter the drill forms into a test or training environment with second-person verification, reconcile against the register, and time the whole exercise. This is the step most drills skip, and it is where most real problems appear.
Keep the drill focused on people and process. Testing whether the vendor can restore your data from backup is a separate exercise with its own evidence. This playbook is about what your staff do while the system is out of reach and how they bring their records back into it.
Measure What Matters
Uptime Institute’s 2026 outage analysis reports that failures to follow established procedures remain the leading driver of human error-related outages.3 The same risk applies to downtime procedures: the plan works only if people follow it. Track a small set of measures from drills and real events:
- Time from first user report to downtime declaration.
- Time from declaration to fallback in use in each area.
- Percentage of downtime forms complete on first review.
- Back-entry error rate found by the second-person check.
- Time from cutover to closed reconciliation.
Review these at periodic review of each critical system and at vendor performance reviews. They give quality and IT leaders an objective view of whether the organization can cope with an outage, rather than a hope that it can.
Common Gaps We See
| Gap | Why It Matters | Fix |
|---|---|---|
| Business continuity described only in the validation plan | Nobody can find or follow it during an outage | A standalone downtime and recovery procedure with a one-page job aid |
| No named decision-maker for continuing GxP work | Work continues or stops by default, not by decision | Named downtime lead and quality decision-maker, each with a backup |
| Downtime forms never updated after system changes | Forms miss fields the system now requires | Review downtime forms under change control when the system configuration changes |
| Recovery not covered in training or drills | Back-entry errors and missing second-person checks | Include a timed recovery exercise in every drill |
| Vendor contract silent on incident notification and root cause reports | Your incident record depends on goodwill | Add notification, evidence, and data access terms at the next renewal |
Conclusion
A validated SaaS outage tests something validation documents rarely show: whether the organization can keep GxP work under control when the system that enforces that control is gone. The regulatory expectations are already clear. Annex 11 asks for documented, tested alternative arrangements with activation times based on risk. MHRA asks for continuity in the contract and restoration to the validated state. PIC/S and FDA make sure downtime paper is treated as a real record. ICH E6(R3) asks for contingency procedures for the data that matters most to trial participants and outcomes. What is usually missing is the operating detail: who declares, who decides, which fallback applies to which process, how the forms look, and how the records come back into the system afterward.
Sakara Digital works with pharma and biotech organizations on the practical side of computerized system compliance, including business continuity for cloud and SaaS systems, vendor agreements, and downtime procedures an inspector can follow from start to finish. If you are reviewing how your organization would handle its first hour without a critical GxP system, and want an independent view of where to start, we are happy to have that conversation.
For Further Reading
References & Sources
- Amazon Web Services. “Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region.” Post-event summary, October 2025. https://aws.amazon.com/message/101925/
- Weston, David. “Helping Our Customers Through the CrowdStrike Outage.” Official Microsoft Blog, July 20, 2024. https://blogs.microsoft.com/blog/2024/07/20/helping-our-customers-through-the-crowdstrike-outage/
- Uptime Institute. “Uptime Announces Annual Outage Analysis Report 2026.” Press release, May 13, 2026. https://uptimeinstitute.com/about-ui/press-releases/uptime-announces-annual-outage-analysis-report-2026
- CrowdStrike. “Falcon Content Update Preliminary Post Incident Report.” CrowdStrike Blog, July 2024. https://www.crowdstrike.com/en-us/blog/falcon-content-update-preliminary-post-incident-report/
- Prince, Matthew. “Cloudflare Outage on November 18, 2025.” Cloudflare Blog, November 18, 2025. https://blog.cloudflare.com/18-november-2025-outage/
- American Hospital Association. “AHA Survey: Change Healthcare Cyberattack Having Significant Disruptions on Patient Care, Hospitals’ Finances.” AHA News, March 15, 2024. https://www.aha.org/news/news/2024-03-15-aha-survey-change-healthcare-cyberattack-having-significant-disruptions-patient-care-hospitals-finances
- European Commission. “EudraLex Volume 4, Annex 11: Computerised Systems.” Revision January 2011. https://health.ec.europa.eu/system/files/2016-11/annex11_01-2011_en_0.pdf
- European Commission. “EudraLex Volume 4: Good Manufacturing Practice (GMP) Guidelines.” Accessed September 2026. https://health.ec.europa.eu/medicinal-products/eudralex/eudralex-volume-4_en
- European Commission and PIC/S. “Annex 11: Computerised Systems.” Draft for stakeholder consultation, 2025. https://health.ec.europa.eu/document/download/40231f18-e564-4043-94de-c031f813d38b_en?filename=mp_vol4_chap4_annex11_consultation_guideline_en.pdf
- Medicines and Healthcare products Regulatory Agency. “‘GXP’ Data Integrity Guidance and Definitions.” Revision 1, March 2018. https://assets.publishing.service.gov.uk/media/5aa2b9ede5274a3e391e37f3/MHRA_GxP_data_integrity_guide_March_edited_Final.pdf
- Pharmaceutical Inspection Co-operation Scheme. “PI 041-1: Good Practices for Data Management and Integrity in Regulated GMP/GDP Environments.” July 1, 2021. https://picscheme.org/docview/4234
- International Council for Harmonisation. “ICH E6(R3) Guideline for Good Clinical Practice.” Step 4, January 6, 2025. https://database.ich.org/sites/default/files/ICH_E6%28R3%29_Step4_FinalGuideline_2025_0106.pdf
- U.S. Food and Drug Administration. “Data Integrity and Compliance With Drug CGMP: Questions and Answers. Guidance for Industry.” December 2018. https://www.fda.gov/media/119267/download
- European Commission. “EudraLex Volume 4, Chapter 4: Documentation.” Revision 2011. https://health.ec.europa.eu/system/files/2016-11/chapter4_01-2011_en_0.pdf
- Office of the National Coordinator for Health Information Technology. “SAFER Self-Assessment: Contingency Planning.” SAFER Guides, 2025 edition. https://www.healthit.gov/wp-content/uploads/2025/06/SAFER-Guide-2.-Contingency-Planning-Final.pdf
- Nelson, A., Rekhi, S., Souppaya, M., and Scarfone, K. “NIST SP 800-61r3: Incident Response Recommendations and Considerations for Cybersecurity Risk Management: A CSF 2.0 Community Profile.” National Institute of Standards and Technology, April 2025. https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-61r3.pdf
- Larsen, E., Hoffman, D., Rivera, C., Kleiner, B.M., Wernz, C., and Ratwani, R.M. “Continuing Patient Care during Electronic Health Record Downtime.” Applied Clinical Informatics, 2019. https://pmc.ncbi.nlm.nih.gov/articles/PMC6620179/
- Lyon, R., Jones, A., Burke, R., and Baysari, M.T. “What Goes Up, Must Come Down: A State-of-the-Art Electronic Health Record Downtime and Uptime Procedure in a Metropolitan Health Setting.” Applied Clinical Informatics 14(3), 2023. https://pmc.ncbi.nlm.nih.gov/articles/PMC10322225/








Your perspective matters—join the conversation.