Why a Validated SaaS Outage Is a Quality Event

When an on-premises system failed ten years ago, the people who could fix it usually worked down the hall. The quality team could walk over, ask what broke, and get a straight answer about whether data was at risk. With software as a service (SaaS), that conversation now runs through a vendor support portal, a status page, and a support engineer in another time zone who may know less about your configuration than your own system administrator does.

The regulatory position has not moved with the hosting model. The regulated company is still responsible for the integrity of its GxP records, for the continuity of the processes those records support, and for showing an inspector what happened. Handing the infrastructure to a vendor moves the work of keeping the system running. It does not move the accountability for what your people do when it stops.

Outages Happen Even When Your Vendor Does Nothing Wrong

A SaaS application depends on layers you never see: a cloud provider, a content delivery network, an identity provider, and the endpoint software on your own laptops. Any one of them can take a validated system out of reach for your users, and several large recent events show how.

14.5 hrsSpan of the October 19 to 20, 2025 AWS disruption in the N. Virginia (us-east-1) Region, from 11:48 PM to 2:20 PM PDT, with three distinct periods of customer impact1
8.5MWindows devices Microsoft estimated were affected by the faulty CrowdStrike update in July 20242
~2/3Share of publicly reported outages that Uptime Institute has attributed to third-party IT and data center service providers3

AWS traced its October 2025 event to a latent defect in the automated system that manages DNS records for its DynamoDB database service. Customers and other AWS services that depended on DynamoDB could not open new connections, and the effects spread to load balancers and new server launches over the following hours.1 Any SaaS product built on that region could have gone down without a single change to its own code.

In July 2024, CrowdStrike released a faulty content update to its Windows sensor at 04:09 UTC and reverted it at 05:27 UTC, 78 minutes later.4 Machines that had already received it crashed, and Microsoft posted manual remediation instructions and scripts to help customers recover.2 For a GxP user, the SaaS system itself may have been fine. The workstation used to reach it was not. From the lab bench, that is still an outage.

Cloudflare’s November 18, 2025 incident began at 11:20 UTC. Core traffic was largely flowing again by 14:30, and all systems were normal by 17:06.5 Two details from Cloudflare’s own write-up matter for the first hour of any outage. The team first suspected a large attack, which turned out to be wrong. And Cloudflare’s status page, hosted entirely outside Cloudflare, went down at the same time by coincidence, which pushed some of the team further toward the wrong explanation.5 If the provider’s own engineers can misread the first signals, your users and your vendor’s first support reply can too.

What Changes When the System Is Validated

For an ordinary business application, an outage is a productivity problem. For a validated GxP system, it raises questions the quality unit must answer:

  • Can the GxP activity continue at all, or must it stop until the system returns?
  • If it continues, where is the official record created, and who controls that record?
  • Were any records lost, partly written, or duplicated when the system failed?
  • Did the vendor’s recovery change the system in a way that affects its validated state?
  • Do any regulatory clocks, such as safety reporting or batch disposition timelines, keep running while the system is down?

None of those questions can be answered well by someone improvising at 7:15 on a Monday morning. They need to be answered in advance, written into a procedure, and practiced. That is the point of a first-hour playbook.

The Scale of the Problem When Workarounds Fail

Health care gives the clearest public evidence of what happens when a critical third-party service stays down for a long time. After the February 2024 cyberattack on Change Healthcare, an American Hospital Association survey of about 1,000 hospitals found that 94% were experiencing a financial impact, and 74% reported impacts to direct patient care. Most hospitals said their workarounds were very labor intensive and costly.6 Pharma operations are different, but the lesson carries over: a workaround that nobody designed in advance is slow, error-prone, and hard to unwind afterward.

What Regulators Expect When a GxP System Goes Down

There is no single regulation titled “SaaS outage.” The expectations are spread across computerized system, documentation, data integrity, and clinical guidance. Read together, they are consistent and fairly specific.

EU GMP Annex 11: Business Continuity and Incident Management

The current EU GMP Annex 11, in force since 2011, has a dedicated business continuity clause. Section 16 says that for computerized systems supporting critical processes, “provisions should be made to ensure continuity of support for those processes in the event of a system breakdown (e.g. a manual or alternative system).” It adds that the time needed to bring the alternative arrangements into use “should be based on risk” and appropriate for the system and business process, and that these arrangements “should be adequately documented and tested.”7

Three other Annex 11 clauses shape what happens during and after the outage. Section 13 says all incidents, not only system failures and data errors, should be reported and assessed, and that the root cause of a critical incident should form the basis of corrective and preventive actions. Section 6 says that for critical data entered manually, there should be an additional check on the accuracy of the data, done by a second operator or by validated electronic means. And Section 3.1 requires formal agreements with third parties, including service providers, with clear statements of their responsibilities.7 Section 6 matters most after recovery, when paper records are keyed back into the system.

A revised Annex 11 was published as a consultation draft in 2025. As of late September 2026, the European Commission’s EudraLex Volume 4 page still lists the January 2011 revision as the current Annex 11.8 The draft does not carry a standalone business continuity chapter. Instead, it asks for a tested disaster recovery plan that, where applicable, ensures continuity of operation within a defined Recovery Time Objective (RTO), and it spells out what contracts with service providers should cover, including incident reporting, SLAs and KPIs, answer times, and resolution times.9 Plan against the 2011 text, which is the one in force, and use the draft as a signal of where inspector expectations are heading.

MHRA and PIC/S: Put Continuity in the Contract

The MHRA data integrity guidance speaks directly to cloud and outsourced services. It says the responsibilities of the contract giver and acceptor should be defined in a technical agreement or contract, and states: “Business continuity arrangements should be included in the contract, and tested.” It also says that “Appropriate arrangements must exist for the restoration of the software/system as per its original validated state, including validation and change control information to permit this restoration.”10 That second sentence is the regulatory basis for the “restore the validated state” step later in this article.

PIC/S guidance PI 041-1 asks companies to give consideration to systems with a higher potential for disaster, malfunction, or situations in which the system becomes inoperative, when deciding the extent of controls.11 Its section on paper documents is just as important for downtime, because downtime paper is still a GMP record. We return to it below.

ICH E6(R3): Contingency Procedures for Clinical Systems

For clinical trial systems, ICH E6(R3) section 4.3.6, titled System Failure, says contingency procedures “should be in place to prevent loss or lack of accessibility to data essential to participant safety, trial decisions or trial outcomes.” The sponsor responsibilities in section 3.16.1 also list backup and disaster recovery among the computerized system requirements a sponsor must make sure are addressed.12 E6(R3) names randomization, dosing, and collection of endpoint data as examples of critical functionality, which gives a useful short list of trial functions whose downtime procedure must be written and tested.12

FDA and EU Documentation Rules Apply to Downtime Paper

FDA’s data integrity guidance is blunt about informal notes: “It is not acceptable to record data on pieces of paper that will be discarded after the data are transcribed to a permanent laboratory notebook.”13 EU GMP Chapter 4 says records should be made or completed at the time each action is taken, and that “It should be clearly defined which record is related to each manufacturing activity and where this record is located.”14 During an outage, those two rules together mean the downtime form is the original record, it must be controlled, and it must be kept after its contents are keyed into the system.

SourceWhat It Asks ForWhere It Applies in the Outage
EU GMP Annex 11, Section 16 (2011)Documented, tested manual or alternative arrangements, with activation time based on riskChoosing and activating the fallback
EU GMP Annex 11, Sections 6 and 13Second check of manually entered critical data; incidents reported, assessed, and root-causedBack-entry and the incident record
Draft Annex 11 (2025 consultation)Tested disaster recovery plan with a defined RTO; contracts covering incident reporting, answer and resolution timesVendor agreement and communication
MHRA data integrity guidance (2018)Business continuity in the contract and tested; restoration to the original validated stateVendor agreement; restoring the validated state
PIC/S PI 041-1Controlled issue and reconciliation of blank formsDowntime paper forms
ICH E6(R3), 4.3.6 and 3.16.1Contingency procedures for data essential to participant safety and trial outcomesClinical systems (EDC, IRT, eCOA, safety)
FDA data integrity Q&A (2018); EU GMP Chapter 4No discarded transcription notes; records made at the time of the action and located in a defined placeRecording data during downtime

The practical reading: regulators do not expect a SaaS system never to fail. They expect you to have decided in advance what happens when it does, to have tested that decision, and to be able to show that every GxP record created during the outage is attributable, contemporaneous, complete, and traceable into the system afterward.

Minutes 0 to 15: Confirm, Declare, and Assign

The first fifteen minutes decide whether the next several hours are controlled or chaotic. The goal is not to fix anything. It is to establish that there is an outage, name the people in charge, and stop anyone from creating records in places nobody will find later.

Confirm the Scope Quickly, Without Waiting for the Vendor

The first report usually comes from a user: a login that hangs, a save that fails, a page that times out. The person receiving it, often the service desk or the system administrator, should run a short, pre-written check:

  • Is it one user, one site, or everyone? Try from a second machine and a second network.
  • Is it the whole application or one module, such as reporting or e-signature?
  • Is the vendor’s status page showing an incident? Treat it as one input, not proof either way.
  • Are other SaaS tools that share the same identity provider also failing? If so, the problem may be sign-in, not the application.
  • Is the system slow rather than down? A system that takes three minutes to save a record can be as disruptive as one that is offline.

The last point is easy to skip. The ONC SAFER guide on contingency planning, written for electronic health records, explicitly calls for functional system downtimes, meaning unacceptably slow response time, to be identified and addressed proactively.15 Your procedure should define what “unacceptably slow” means for each critical system, so nobody has to argue about it in the moment.

Declare Downtime Against Written Criteria

NIST’s incident response guidance, SP 800-61 Revision 3, recommends that incidents be declared when adverse events meet defined incident criteria.16 The same principle works for GxP downtime. Write the criteria down in advance: for example, a critical system unavailable or unusably slow for a set number of minutes during working hours, or any sign that data may have been lost or corrupted. When the criteria are met, the downtime lead declares downtime. They do not wait for the vendor to confirm, because the vendor may take an hour to acknowledge the incident, and the GxP work in front of your people cannot wait that long without a decision.

Declaring downtime and later finding out it was a short blip is a minor inconvenience. Not declaring it, and letting staff improvise for two hours, creates records you will spend weeks reconstructing.

Who Decides: Four Named Roles

Annex 11 already expects close cooperation among the process owner, the system owner, qualified persons, and IT, with defined responsibilities.7 For downtime, turn that into four named roles, each with a named backup:

Leads

Downtime Lead (Business Process Owner)

Declares downtime, decides whether the business process pauses or moves to the fallback, and declares the return to normal. Owns the business decision, not the technical fix.

Approves

Quality Decision-Maker

Agrees that GxP work may continue under the fallback, sets any limits (for example, no batch disposition until the system returns), and opens the deviation or incident record.

Liaises

System Owner or IT Liaison

The single point of contact with the vendor. Opens the ticket, logs every exchange with times, and translates vendor updates into plain facts for the downtime lead.

Informs

Communications Lead

Tells affected users what is happening, what to do, and when the next update will come. Uses channels that do not depend on the failed system or its sign-in.

A common failure: the system owner is also the person expected to decide whether manufacturing or lab work continues. In the first hour, that person is busy with the vendor. Separate the technical liaison from the business decision, and make sure the quality decision-maker is a named person with authority, not “QA” as a group.

Freeze Uncontrolled Record-Keeping

The last step in the first fifteen minutes is a short instruction to all affected users: stop entering data into the system, and do not record GxP data on sticky notes, in personal spreadsheets, in email, or in chat messages. Tell them where the downtime kit is and who will tell them what to do next. This instruction prevents a common downtime problem: data that exists only in an informal place and cannot be shown to be complete.

Minutes 15 to 30: Choose the Fallback

Annex 11 describes the alternatives as “a manual or alternative system.”7 In practice there are four options, and a mature playbook has already chosen among them for each critical process, so the downtime lead is confirming a plan rather than designing one.

Option 1: Pause the Activity

Many GxP activities can wait. Approving a change control, closing a CAPA, releasing a controlled document, or signing off a validation report rarely has to happen in the next four hours. For these, the safest fallback is to stop and resume in the system when it returns. Pausing creates no hybrid records and no back-entry, and nothing needs to be reconciled. The playbook should say which activities are “pause by default,” so nobody feels pressure to invent a paper approval process in the moment.

Option 2: Controlled Paper

Where the work cannot stop, such as an in-process manufacturing step, a time-sensitive lab test, or a clinical site visit, controlled paper forms are the usual fallback. “Controlled” is the key word. These are pre-approved downtime forms that capture the same critical data as the system, issued and reconciled under the same rules as any other GMP form. The next section covers how to design and use them.

Option 3: An Alternate System or Read-Only Copy

Some companies keep a validated alternate: a local data store on an instrument, a secondary system that can take the work temporarily, or a read-only copy of key data. The SAFER guide recommends a read-only backup for electronic health records that is tested regularly and allows users to print, so clinicians can see current information even when the main system is down.15 In pharma, the equivalent is often a regularly refreshed, controlled export of the information people need to keep working safely: current specifications, open batch status, active study subject lists, or current effective SOPs.

Option 4: Continue in the Parts That Still Work

Partial outages are common. The review module may be down while data capture works, or the web client may fail while a mobile app still connects. Continuing in the working parts can be sensible, but only if the playbook has already confirmed that the working module’s records are complete on their own and that no required control, such as an audit trail or e-signature, depends on the failed component.

System TypeTypical First-Hour FallbackWhat to Watch
eQMS and document managementPause approvals; use controlled, time-stamped printed copies of current SOPsOperators need current procedures; printed reference copies must show when they were printed and how long they are valid
LIMSControlled paper worksheets; instrument data held in the instrument’s own storageSample identity, chain of custody, and instrument data that must later be linked to the right sample
MES or electronic batch recordPre-approved paper batch record sections for the affected stepsCritical process parameters, second-person verification, and a clear point where the paper takes over and hands back
EDC, eCOA, or IRT (clinical)Site source documents; emergency randomization or dosing procedure agreed in advanceRandomization, dosing, and endpoint data are named as critical functionality in ICH E6(R3)
Safety databaseControlled case intake logReporting clocks continue; case receipt dates must be captured accurately

How Fast the Fallback Must Start

Annex 11 leaves the activation time to risk: it must be appropriate for the system and the business process it supports.7 Health care offers useful reference points. The SAFER guide recommends paper forms on hand for at least 8 hours of care in each area, and a policy that says when a warm-site backup should be activated, ideally before an unplanned outage reaches 2 hours.15 A pharma company will set its own numbers, but it should set them. “We will switch to paper after 30 minutes of downtime on the packaging line, and after four hours for the document system” is a decision. “We will assess the situation” is not.

Minutes 30 to 60: Recording GxP Data During Downtime

By the half-hour mark, users should be working from the fallback. The quality of the records they create now decides how hard the recovery will be. The evidence from health care is sobering. In a study of downtime at a 300-bed hospital, researchers found laboratory results were delayed by an average of 62% during downtime, and the archived data were incomplete due to inconsistencies in the downtime paper records. Staff interviews showed the delays were often due to improper procedural execution and incomplete or incorrect documentation.17 Paper under pressure tends to be incomplete unless the form and the process make completeness easy.

62%Average delay in laboratory results during EHR downtime in one hospital study, with incomplete downtime paper records limiting the analysis17
8 hrsPaper forms the SAFER guide recommends keeping on hand in each area, with a documented process to enter and reconcile them afterward15
6 hrsTime one health district needed to complete recovery and reconciliation after an 8-hour planned downtime18

Design Downtime Forms That Mirror the System

A good downtime form asks for exactly what the system would have captured, in the same order, with the same units and the same required fields. Each form should include:

  • A unique form number, issued from a register, so the set can be reconciled later.
  • The downtime event reference, so every form links to one incident record.
  • Fields for the actual date and time of each observation or action, taken from a defined reference clock.
  • The identity of the person doing the work, and of any second person verifying it.
  • Fields for the identifiers the system would normally generate or check, such as sample IDs, lot numbers, or subject numbers.
  • A block for the back-entry step: who entered it, when, and who verified the entry.

PIC/S PI 041-1 sets out the controls for blank forms, and they apply fully to downtime forms: record who issued copies and when, allocate a unique identifier to each blank document and record its issue in a register, and reconcile critical GMP blank forms after use to ensure the records are accurate and complete.11 The downtime kit should hold pre-numbered forms and the register, stored where people can reach them without the failed system.

Keep the Paper as the Original Record

When a user writes a result on a controlled downtime form, that form is the original record. EU GMP Chapter 4 expects records to be made at the time each action is taken.14 FDA says it is not acceptable to record data on paper that will be discarded after transcription.13 So the paper is retained, linked to the electronic entry, and treated as part of a hybrid record for the life of the data. Nobody should shred downtime forms once “everything is in the system.”

Signatures, Approvals, and Printed SOPs

If an approval must happen during downtime and cannot wait, it is a handwritten signature on the controlled form, dated with the actual date. When the system returns, do not re-create that approval as an electronic signature dated as if it happened during the outage. Record in the system that the approval was given on paper, reference the form, and attach a scan if your procedure allows. The system’s audit trail will show the true entry time, and your records should agree with it.

If the document management system is down, operators may lose access to current SOPs. PI 041-1 allows printed reference copies of documents that are not records without reconciliation, provided they are time-stamped when generated and their short-term validity is marked on the document.11 A controlled, regularly refreshed set of current SOPs for critical operations, with a printed-on date and a “valid until” date, is one of the most useful items in any downtime kit.

What inspectors look for after an outage: GxP data that appears in the system with no matching downtime form; forms that are missing from the register; back-entered records dated to the original activity time with no note explaining the late entry; results transcribed without a second-person check; and approvals that seem to have been given electronically while the system was down.

Keep a Running Log

The downtime lead should keep a timed log from the moment of declaration: when the fallback started in each area, which forms were issued, what the vendor said and when, and every decision taken. This log becomes the main source for the incident record. It also answers the question an inspector may ask months later: what happened that day, in what order, and who decided what.

Working With the Vendor During the Outage

The vendor controls the repair. Your job is to get the facts you need to make quality decisions and to hold the vendor to the commitments in your agreement.

One Voice, One Ticket, One Log

Only the named system owner or IT liaison should talk to the vendor about the incident. When five users open five tickets, the vendor’s support team spends time merging them, and your company receives five slightly different answers. The liaison opens one priority ticket, gives the incident reference, and logs every exchange with the time it happened. NIST recommends following the rules and protocols defined in contracts for incident information sharing between the organization and its suppliers.16 That only works if your contract has such rules, which is a job for procurement and quality long before any outage.

What to Ask in the First Call

Vendors’ first responses tend to be general: “We are aware of an issue and are investigating.” Push, politely, for the answers that affect your quality decisions:

  1. Scope. Is the incident affecting all customers, one region, or only your tenant? Is it the application, the sign-in service, or an underlying cloud provider?
  2. Data at risk. Were any transactions in progress lost or partly written? If the vendor restores or fails over, what is the recovery point, meaning the last moment from which data is guaranteed?
  3. Recovery approach. Will the vendor fail over to a secondary site, roll back a recent change, or apply a fix? Each of these has different implications for your validated state.
  4. Next update. When will the next update come, even if there is no news? A fixed update time lets your communications lead set expectations.
  5. Root cause report. Will there be a written post-incident report, and when? You will need it for your incident record.

Treat Early Information as Provisional

The Cloudflare example shows why. Its engineers first suspected an attack, and a coincidental failure of the separately hosted status page reinforced that wrong theory.5 Early vendor updates may be similarly wrong. Record them, act on what is certain, and do not make an irreversible quality decision, such as rejecting a batch or discarding a sample, based on a vendor’s first guess about the cause.

What the Contract Should Already Say

The MHRA expects business continuity arrangements in the contract, and tested.10 The draft Annex 11 lists what a service contract should agree on, including incident reporting and oversight, SLAs and KPIs, answer times, and resolution times, as well as support during regulatory inspections.9 When you negotiate or renew a GxP SaaS agreement, check that it covers:

  • How and how fast the vendor notifies you of an incident affecting your tenant, rather than relying on you to notice.
  • The vendor’s recovery time and recovery point commitments, and how they are tested.
  • Whether a failover or emergency fix is performed under the vendor’s change control, and what evidence you receive.
  • A commitment to provide a written root cause report within a set time.
  • Access to your data, including audit trails, during and after an incident.

An SLA credit for missed availability is useful to procurement, but it does nothing for your records. The terms that matter for quality are notification, evidence, and data access.

After Recovery: Reconciling Data and Restoring the Validated State

When the vendor reports that service is restored, the temptation is to tell everyone to log back in and carry on. Resist it. The return to normal needs as much control as the switch to downtime, because this is where most data integrity problems are created.

Verify Before You Resume

NIST recommends verifying the correctness and adequacy of restoration actions before putting a restored system online, and working with system owners to confirm successful restoration and the return to normal operations.16 For a validated SaaS system, a short, pre-written post-recovery check might cover:

  • Users can sign in with their normal accounts and roles, and access rights are as they were.
  • The audit trail is on and recording, with correct time stamps and time zone.
  • Electronic signatures work and are linked to their records.
  • The last records entered before the outage are present and complete.
  • Interfaces to instruments, ERP, or other systems are running, and any queued messages are accounted for.
  • Key configuration settings, such as workflows, required fields, and calculation settings, match the approved configuration.

This is a targeted check, not a revalidation. Its purpose is to confirm that the system you are going back into is the system you validated.

Declare a Single Cutover Point

The Lyon and colleagues account of a health district’s downtime and recovery procedure describes two useful rules. The recovery phase starts only when the system is formally announced ready, and staff keep prescribing and recording medication administration on paper “until reconciliation has been completed.”18 The same logic applies to GxP work. The downtime lead announces one cutover time. Before it, the paper is the record. After it, the system is. Any activity in progress at the cutover finishes on paper and is back-entered with the rest.

Back-Enter With a Second-Person Check

Back-entry is transcription of critical data, so Annex 11 Section 6 applies: an additional accuracy check by a second person or validated electronic means.7 Set up the back-entry so that:

  • Each entry references its downtime form number.
  • The actual activity date and time are recorded in the proper fields, and the system’s own entry time stamp is left as it is. The two times will differ, and they should.
  • A comment or reason code marks the record as a late entry from downtime, where the system allows it.
  • A second person verifies each critical entry against the form and records the verification.
  • Scans of the forms are attached or linked, if your procedure uses scanning.

Plan the effort honestly. The same health district found it needed 2 to 3 hours for a doctor and pharmacist pair to reconcile medication charts in a high-use area, and 30 to 45 minutes in lower-use areas. It finished recovery within 6 hours after an 8-hour planned downtime.18 Those numbers will not transfer directly to a QC lab or a packaging line, but the lesson does: reconciliation takes real time and trained people, and the playbook should estimate it for each area.

Reconcile the Whole Set, Not Just the Forms You Have

Reconciliation should answer four questions, and the answers belong in the incident record:

1

Are All Issued Forms Accounted For?

Compare the form register with the forms returned. Every issued form is either completed and entered, voided with a reason, or returned unused.

2

Is Every Completed Form in the System?

Check that each completed form has a matching back-entered record, verified by a second person.

3

Did Anything Happen Outside the Forms?

Look for activity that should have produced a form but did not: instrument runs with no worksheet, samples received with no log entry, users who kept working in a module that still responded.

4

What Did the System Lose or Duplicate?

Check records in progress at the moment of failure, interface messages that were queued and replayed, and any gap between the vendor’s recovery point and the outage.

The fourth question needs particular care. If the vendor recovered from a backup or failed over to a secondary site, anything written between the recovery point and the failure may be gone. Those records were never on paper, because nobody knew the system had failed yet. Find them by comparing the last records users remember entering with what the system now shows, and by checking instrument logs and interface message queues. Re-enter lost records from their original sources, with a note explaining why. Duplicates from replayed interface messages need the opposite treatment: identify them, then correct them under your normal data correction procedure with a documented reason.

Decide Whether the Validated State Is Intact

The MHRA expects arrangements to restore the system to its original validated state, including the validation and change control information needed to do so.10 After a SaaS outage, the quality decision rests on the answer to one question: did the recovery change the system?

  • No change to your system. The cause was upstream, such as a network or cloud provider failure, and the vendor restored service without changing the application or your configuration. The post-recovery check is usually enough, documented in the incident record.
  • A vendor fix or rollback. The vendor deployed a hotfix or reverted a release. Treat it as an unplanned change. Get the vendor’s description of what changed, assess its impact against your validated functions, and run targeted verification of the affected functions under your change control.
  • Failover or restore. Your tenant now runs from a secondary site or from restored data. Confirm with the vendor that the failover was performed under their qualified disaster recovery process, confirm your configuration and audit trails came across intact, and reconcile any data lost between the recovery point and the failure.

Exit criteria for closing the outage: post-recovery check completed and signed; cutover time recorded; all downtime forms reconciled against the register; all back-entries verified by a second person; lost and duplicate records identified and corrected; vendor root cause report received and assessed; validated state decision documented by quality; and CAPA opened where the root cause or your own response showed a gap.

Close the Incident Properly

Annex 11 Section 13 asks for every incident to be reported and assessed, and for the root cause of a critical incident to form the basis of CAPA.7 For a SaaS outage, the root cause of the system failure usually belongs to the vendor, and your record should include their report. But your own response has its own root causes. Did it take 90 minutes to declare downtime because nobody knew the criteria? Were forms missing from the kit? Did the call tree depend on the same sign-in service that failed? Those are your gaps to fix. NIST recommends closing recovery with an after-action report covering the incident, the response and recovery actions, and lessons learned.16 The SAFER guide goes further for long events, recommending an in-depth review of unexpected downtimes lasting over 24 hours using root cause or failure modes and effects analysis.15

Building the Playbook Before You Need It

Everything in the first hour depends on work done months earlier. A playbook that exists only as a paragraph in a validation plan will not help at 7:15 on a Monday. Here is what a working one contains.

Start From a Tiered System List

List every GxP SaaS system and rank it by how long each supported process can tolerate the system being unavailable. That tolerance, not the vendor’s SLA, sets your activation time. A packaging line that needs an electronic batch record may tolerate 30 minutes. A training records system may tolerate several days. Record the answer for each system, with the fallback option chosen for each critical process. The SAFER guide also recommends keeping a list of system-to-system interfaces, reviewed regularly, with a note of any that carry legal or regulatory notification duties if they go down.15 Clinical and safety systems are the obvious candidates for that note in pharma.

Assemble the Downtime Kit

For each critical process, the kit holds:

  • Pre-approved, pre-numbered downtime forms and the issue register.
  • A one-page job aid: how to declare, who to call, what not to do.
  • Current printed reference copies of the SOPs needed to keep working, with printed-on and valid-until dates.
  • The contact list and call tree, on paper.
  • The downtime and recovery procedure itself, on paper.

The SAFER guide recommends a paper copy of the downtime and recovery policy in each clinical unit and another off site, with the policy reviewed at least every 2 years.15 The reasoning holds for a GMP site. A procedure stored only in the system that just failed is no procedure at all.

Make Communication Independent of the Failed System

The SAFER guide recommends that downtime communication not rely on the same computing infrastructure as the failed system, and names email, websites, and voice-over-IP as examples of channels that may fail alongside it. It suggests a mobile phone-based call tree instead.15 Many pharma companies now run email, chat, and their GxP SaaS systems behind one identity provider. If sign-in fails, all of them fail together. Test whether your downtime communications still work when single sign-on does not.

Drill It, Including the Recovery

Annex 11 requires the alternative arrangements to be documented and tested.7 The SAFER guide recommends unannounced downtime drills at least once a year, and training that covers both downtime and recovery procedures.15 A useful GxP drill has three parts:

1

Tabletop Walk-Through

The four named roles talk through a realistic scenario, such as a vendor-side outage during a batch or a sign-in failure during a site monitoring visit, using the written procedure. Gaps in decision rights show up quickly.

2

Hands-On Downtime Exercise

Users in one area complete a set of realistic entries on downtime forms, drawn from the kit, under time pressure. Check the forms for completeness afterward.

3

Recovery Exercise

Back-enter the drill forms into a test or training environment with second-person verification, reconcile against the register, and time the whole exercise. This is the step most drills skip, and it is where most real problems appear.

Keep the drill focused on people and process. Testing whether the vendor can restore your data from backup is a separate exercise with its own evidence. This playbook is about what your staff do while the system is out of reach and how they bring their records back into it.

Measure What Matters

Uptime Institute’s 2026 outage analysis reports that failures to follow established procedures remain the leading driver of human error-related outages.3 The same risk applies to downtime procedures: the plan works only if people follow it. Track a small set of measures from drills and real events:

  • Time from first user report to downtime declaration.
  • Time from declaration to fallback in use in each area.
  • Percentage of downtime forms complete on first review.
  • Back-entry error rate found by the second-person check.
  • Time from cutover to closed reconciliation.

Review these at periodic review of each critical system and at vendor performance reviews. They give quality and IT leaders an objective view of whether the organization can cope with an outage, rather than a hope that it can.

Common Gaps We See

GapWhy It MattersFix
Business continuity described only in the validation planNobody can find or follow it during an outageA standalone downtime and recovery procedure with a one-page job aid
No named decision-maker for continuing GxP workWork continues or stops by default, not by decisionNamed downtime lead and quality decision-maker, each with a backup
Downtime forms never updated after system changesForms miss fields the system now requiresReview downtime forms under change control when the system configuration changes
Recovery not covered in training or drillsBack-entry errors and missing second-person checksInclude a timed recovery exercise in every drill
Vendor contract silent on incident notification and root cause reportsYour incident record depends on goodwillAdd notification, evidence, and data access terms at the next renewal

Conclusion

A validated SaaS outage tests something validation documents rarely show: whether the organization can keep GxP work under control when the system that enforces that control is gone. The regulatory expectations are already clear. Annex 11 asks for documented, tested alternative arrangements with activation times based on risk. MHRA asks for continuity in the contract and restoration to the validated state. PIC/S and FDA make sure downtime paper is treated as a real record. ICH E6(R3) asks for contingency procedures for the data that matters most to trial participants and outcomes. What is usually missing is the operating detail: who declares, who decides, which fallback applies to which process, how the forms look, and how the records come back into the system afterward.

Sakara Digital works with pharma and biotech organizations on the practical side of computerized system compliance, including business continuity for cloud and SaaS systems, vendor agreements, and downtime procedures an inspector can follow from start to finish. If you are reviewing how your organization would handle its first hour without a critical GxP system, and want an independent view of where to start, we are happy to have that conversation.

For Further Reading