Small Language Models On-Prem: When Pharma Should Not Use a Frontier API

The Default Is Usually Right, and That Is the Problem

Sit in enough architecture reviews at pharma and biotech companies and you notice a pattern. Someone proposes a use case. Someone else says the data is sensitive. A third person says we should run our own model. Nobody in the room has costed it, nobody has tested whether a smaller model can do the task, and nobody has asked which specific rule or contract would be violated by the API call. The meeting ends with an action item to “explore on-prem options” and six months later there is a server in a rack running a model that two people know how to restart.

The opposite failure also happens. A team assumes the API is fine because procurement signed a data processing agreement, then discovers during a partner audit that the collaboration agreement for a co-development program prohibits disclosure of certain data to any third party, with no carve-out for cloud processors. That is a contract problem, not a security problem, and no amount of encryption fixes it.

Both failures come from the same place: the decision is made on vibes about sensitivity rather than on the specific constraint that actually applies. Sensitivity is not a decision criterion. It is a feeling about a decision criterion. The useful question is narrower and much easier to answer: what specifically prevents this data from reaching this endpoint, and is that thing a law, a contract, a network, a clock, or a validation obligation?

Why the frontier API deserves to be the default

Start from the honest baseline. A frontier model accessed through an enterprise API gives you capability that no 7B or 14B open-weight model matches on open-ended work, and it gives you that capability without a capital request, a hiring plan, or a data center conversation. The provider handles model serving, scaling, availability, and the security of the inference stack. Enterprise agreements now routinely include commitments that inputs and outputs are not used for model training and are not retained beyond the processing window, a pattern that has become standard enough to be studied in its own right in the literature on enterprise assistant deployments.¹⁷

There is also a quality argument that gets ignored. A major provider runs a security program, patches its serving infrastructure, and carries audited certifications. Most mid-size biotech IT groups cannot match that with the staff they have. Choosing to self-host is choosing to take on a security responsibility that you were previously buying. That trade can be worth making. It is never free.

What the market signal actually says

The pressure toward controlled deployment is real and it is growing, but it is being driven by jurisdiction rather than by capability anxiety. Gartner forecast worldwide sovereign cloud infrastructure-as-a-service spending at $80 billion in 2026, a 35.6% increase over 2025, with governments as the primary buyers followed by regulated industries and critical infrastructure operators.⁴ Europe is forecast to pass North America in sovereign cloud IaaS spending in 2027.⁵ That is a story about who controls data and under whose law, not a story about anyone deciding frontier models are inadequate.

The Five Constraints That Actually Force the Decision

Here are the five that hold up under scrutiny. If none of them applies to a use case, the API is your answer and the conversation should end.

1. Data residency and sovereignty commitments

Residency and sovereignty are different things and conflating them causes most of the confusion. Residency is about where bytes physically sit. Sovereignty is about whose law can compel access to them. A provider region inside the EU satisfies the first. It does not, on its own, satisfy the second, because a provider incorporated in a third country may remain subject to that country’s compulsory process regardless of where the servers are.

The European Data Protection Board addressed the substance of this directly in its recommendations on supplementary measures for international transfers, which set out that contractual measures alone cannot cure exposure to third-country access powers, and that technical measures are required where the importer can be compelled to disclose.⁶ The EDPB also treats remote access from a third country, including access for support purposes, as a transfer in its own right. That is the detail that catches teams out: your data never left the EU, but a support engineer in another jurisdiction could read it, and that is a transfer.

For a pharma organization, this matters most where a commitment has already been made in writing. A national health system data agreement, a public research grant condition, or a government partnership may commit you to processing inside a specific jurisdiction under specific law. Those commitments are enforceable and they were made by people who are not in your architecture review. Read them before you design.

2. Contractual restrictions from a partner or a clinical site

This is the constraint most often discovered late and the one least amenable to technical fixes. Clinical trial agreements, confidentiality agreements, and co-development contracts frequently restrict disclosure of study materials and participant information to third parties. Access to trial materials does not carry an implied right to route them through an unrelated third-party system, and sites are increasingly being reminded of exactly that as staff put protocols, investigator brochures, source notes, and correspondence into general-purpose tools.¹⁸

Two practical consequences follow. First, a use case that is fine for your own manufacturing records may be prohibited for a partner’s clinical data under the same architecture, because the restriction attaches to the data, not the system. Second, the fix is usually legal rather than technical. Amending an agreement to name a category of processor is often faster and cheaper than building an isolated environment, and it produces a durable answer instead of a one-off workaround.

3. Air-gapped manufacturing environments

This one is physical and it is not negotiable by contract. Plenty of GMP shop floors sit behind a segmented network with no outbound path to the internet, by design and by the site’s own security architecture. NIST’s guidance on operational technology security describes the zone and conduit approach that governs these environments and the strict control of any communication crossing a boundary.⁸ If a batch record review assistant or a deviation triage helper needs to run against systems inside that boundary, an outbound API call is not an inconvenience. It is a change to the plant’s security architecture, and it will be reviewed as one.

Note the honest version of this constraint. Very few sites are truly air-gapped in the strict sense. Most are segmented, with controlled conduits, historians that replicate outward, and data diodes or brokered paths that already move information from the floor to enterprise systems. The real question is whether your use case can live on the enterprise side of an existing, already-qualified boundary, working on data that has already crossed. Often it can, and then you do not need a model on the floor at all. When the use case genuinely has to act at the point of manufacture, you do.

4. Latency at the edge

Some decisions have to happen inside a control loop. If a model output feeds a reject gate on a filling line, a round trip to a cloud endpoint is not a design choice, it is a defect. This constraint is real but it is also narrow, and it usually points to a small purpose-built model rather than a general language model. Reviews of edge AI architecture set out the trade-offs clearly: inference is pushed to the device because of latency, availability, and bandwidth limits, and the resulting systems carry their own management burden in exchange.¹¹

Be careful not to borrow this constraint where it does not apply. A document assistant used by a QA reviewer does not have a latency constraint in any meaningful sense. A few hundred milliseconds is invisible in that workflow. Latency justifies local inference when the consumer of the output is a machine on a clock, not when the consumer is a person reading a screen.

5. The validation burden of a model that changes

The fifth constraint is the one that most deserves a section of its own, so it gets one.

The Most Underrated Constraint: A Model That Changes While Your Validation Does Not

A qualified state is a claim about a specific configuration. You tested a defined system, against defined inputs, and produced evidence that it performs as intended for its context of use. If any part of that system changes without assessment, the claim is no longer supported. This is not a novel idea. It is the ordinary logic of change control applied to a component that happens to be a model.

The uncomfortable fact is that a hosted model can change while your validation package sits untouched in the document management system. A 2025 replication study makes the consequence concrete. Researchers reviewed 85 published papers that used commercial hosted models, found only 18 that supplied research artifacts, found only 5 of those complete enough to actually execute, and then re-ran them. None of the 5 fully reproduced its original results. Two reproduced partially and three did not reproduce at all.³ The point is not that the original researchers were careless. It is that the component they depended on was no longer the same component by the time anyone tried to check their work.

A local weight file that you control does not do that. It is a byte-identical artifact with a hash you can record in your validation package and verify at any time. That is a genuine and durable advantage, and it is the single strongest technical argument for self-hosting in a regulated setting. It is worth stating plainly because it usually gets buried under weaker arguments about security.

How to satisfy the obligation without owning the weights

Three things make a hosted model workable under change control, and they should be procurement requirements rather than hopes.

Pin to a dated snapshot, never to a moving alias. Most providers publish both a floating identifier that always points at the newest release and a dated identifier that does not move. Application code should reference the dated one. This is a one-line change that removes the largest single source of silent behavioral drift.

Negotiate notice and overlap. Ask for a contractual minimum notice period before a pinned version is retired, and a defined overlap window during which the old and new versions are both available. Without overlap you cannot run a comparison, and without a comparison you cannot make a change assessment. A short notice period is a validation problem disguised as a commercial term.

Build the regression suite before you need it. A fixed set of representative inputs with known-good outputs, run on a schedule against the pinned endpoint, is what turns “the model may have changed” from an anxiety into a detection. This is the same discipline as periodic review, applied at a cadence that matches how fast the underlying component can move.

What a Smaller Open-Weight Model Does Well, and Where the Gap Is Still Real

The capability question deserves specificity rather than slogans. Open-weight models in the 7B to 30B range are genuinely useful for a defined class of work and genuinely weaker on another class. Knowing which is which is most of the decision.

Where smaller models hold up

Classification and extraction over a narrow domain. This is the strongest case. A study of privacy-sensitive clinical information extraction using locally deployable small models reported micro and macro F1 of 0.864 and 0.837 for a 14B model and 0.806 and 0.797 for an 8B model after preference optimization, on entity extraction from clinical notes.¹¹¹² A separate 2026 study extracting thirteen clinical features from more than 1,200 anonymized transcripts found a 7B open model reaching median macro F1 of 0.899, and reported the useful nuance that models in the 7B to 8B range consistently outperformed smaller ones on sensitivity.¹² These are usable numbers for a first-pass extraction step with human review behind it.

Constrained generation into a fixed schema. When the output has to conform to a structure, small models do better than their general reputation suggests, because the structure does part of the work. A benchmark of locally run open models generating SQL against a synthetic biopharmaceutical manufacturing database of roughly 63,000 records across batch, MES, and clean-in-place modules found that three of four tested models produced SQL for all sixty evaluation tasks, with a coding-specialized 7B model and an 8B general model performing best on different axes.¹³ The same study is candid that current performance still requires human oversight and downstream validation before regulated use, which is the correct posture.

Text-anchored transformation. A prospective pilot of a self-hosted open-weight model in a hospital radiology setting, deployed with strict network segmentation and host-enforced egress filtering, found that text-anchored tasks such as report correction, simplification, and guideline lookup received the highest utility ratings from the twenty-two clinicians in the pilot.¹⁰ Rewriting, reformatting, summarizing against a source that is present in the prompt: this is where the local option earns its keep.

Where the gap is still real

The same radiology pilot is the clearest published statement of the limit. Open-ended conclusion generation from findings produced the highest frequency of critical errors of any task in the study.¹⁰ That is the pattern in one sentence: anchor the model to text in front of it and it performs; ask it to reason its way to a judgment that is not in the source and error rates rise sharply.

The quantitative gap on harder document work is also measurable. A January 2026 benchmark of information extraction from medical documents reported an average F1 of 55.6 for a hosted proprietary model against 41.3 for the best locally hosted open model tested, a gap of roughly fourteen points on the same task set.¹⁴ That gap is the price of local control on that class of work, and it should be stated as a number in your assessment rather than waved away.

The quantization question

Local deployment usually means quantized weights, and quantization is often discussed as if it were free. The best available evidence says it is close to free at 8 bits and mostly acceptable at 4 bits for larger models, but not uniformly so. A large evaluation across a model family, covering more than 500,000 individual assessments, found FP8 effectively lossless at all scales tested, well-tuned INT8 at 1% to 3% accuracy degradation, and 4-bit weight-only quantization competitive with 8-bit in many settings.¹⁵ The important qualifier for a regulated buyer is that degradation is not evenly distributed across tasks or model sizes, so an average figure tells you nothing about your task. Evaluate the exact quantized artifact you intend to deploy, on your own data, and record that evaluation in the validation package. The quantized file is the system, not an approximation of it.

The Honest Total Burden of Running Your Own

This is where most business cases fall apart, and they fall apart in a specific way: the plan accounts for the hardware and nothing else. Hardware is the smallest and most predictable part.

Add these together and the pattern becomes clear. Organizations routinely underestimate the total effort by a wide margin, because they price the visible item and ignore the standing ones. The security burden alone deserves more weight than it gets. The documented remote code execution flaws across inference frameworks arose from copied deserialization patterns rather than exotic attacks, which means they are the kind of issue that recurs.¹⁶ When you self-host, that patch lands on your change control queue, and it lands on a system that is inside your validated boundary.

The Middle Options Most Organizations Should Land On

The framing of this decision as a binary between a public API and your own server room is the reason so many teams get it wrong. There is a wide middle, and for most pharma and biotech organizations the answer sits there.

Dedicated capacity within a provider environment

Most major providers now offer deployment modes where inference runs on capacity dedicated to your organization rather than shared, sometimes inside your own cloud account or private network boundary. You do not hold the weights, but the traffic does not traverse a multi-tenant service and the network path can be constrained to your own private connectivity. This addresses a real class of security objection and most of the network segmentation concern, without taking on the hardware or MLOps burden.

Regional hosting to satisfy residency

Where the constraint is residency rather than sovereignty, regional deployment is the direct fix and it is straightforward to evidence. Confirm in writing which regions process, which regions store, where any logging and abuse-monitoring data lands, and from which jurisdictions support staff can access the environment. That last point is the one teams forget, and the EDPB treats it as a transfer in its own right.⁶ If the constraint is sovereignty rather than residency, regional hosting alone will not close it, and you need to look at sovereign offerings with separate legal entities and locally resident operations, or at self-hosting.

Contractual commitments that make a hosted model auditable

The commercial terms do more work here than any technical control. A short list worth putting into a schedule rather than an email:

• Version pinning. Named dated model versions, with a commitment that a pinned version does not change behavior during its supported life.
• Deprecation notice and overlap. A minimum notice period before retirement and a defined window where the outgoing and incoming versions are both callable, so you can run a comparison.
• Data use and retention. No training on your inputs or outputs, retention limited to the processing window, and a description of what happens to any abuse-monitoring copies.¹⁷
• Processing locations and subprocessors. Named regions, a current subprocessor list, and notice of change.
• Support access. Which staff, in which countries, can access which data, and under what controls.
• Documentation for downstream obligations. Where the EU AI Act applies, providers of general-purpose models carry documentation obligations toward downstream providers who build on them, and you should be receiving that documentation rather than reconstructing it.⁷
• Audit and evidence. Right to receive certifications, test evidence, and incident notifications on a defined timeline.

A Decision Framework Keyed on the Binding Constraint

Frameworks that score twenty criteria produce a number that nobody trusts. This one asks which single constraint is binding, because that is what determines the answer. Work through it per use case, not per organization. The same company will land in different places for a manufacturing vision system, a regulatory writing assistant, and a pharmacovigilance triage step, and that is correct.

What This Looks Like in an Architecture

The practical result of applying that framework is almost never a single choice. It is a small number of placements, each justified by a named constraint, sitting behind a common interface.

Route by constraint, not by preference

Put an internal gateway between your applications and whatever serves them. The gateway holds the routing rules, and the rules are expressed in terms of data classification and constraint, not in terms of which model somebody likes. A request carrying partner clinical data under an absolute disclosure bar routes to the internal deployment. A request carrying internal SOP text routes to the pinned hosted version. The application code does not change when a contract changes; the routing rule does.

Keep one evaluation harness for all placements

The most common architectural mistake is building separate quality processes for the hosted and local paths. Run the same evaluation set against every placement on the same schedule. This gives you a like-for-like comparison, which is what you need when someone asks whether the local model is still good enough, or whether the hosted one has moved. It also means that when a constraint disappears, you can move the workload with evidence rather than argument.

Design the task so the smaller model can succeed

Where a constraint forces local inference, the highest-value engineering work is usually not model selection. It is task decomposition. Break an open-ended request into an anchored retrieval step, a constrained extraction step, and a deterministic assembly step. The published evidence is consistent that anchored and constrained work is where smaller models perform and open-ended synthesis is where they fail.¹⁰¹³ A well-decomposed task on a 7B model will beat a badly framed one on a much larger model, and it will be far easier to validate because each step has a checkable output.

Write the qualification package around the boundary, not the model

Whichever placement you choose, the validated system includes the retrieval layer, the prompt templates, the output parsers, the guardrails, and the human review step. The model is one component inside that boundary. Teams that write their validation package around the model alone end up unable to explain what changed when a retrieval index is rebuilt or a prompt template is edited. Define the boundary first, then place the model inside it.

Conclusion

The frontier API is the right default for pharma and biotech, and most of the arguments made against it are arguments from discomfort rather than from a named constraint. What changes the answer is specific: a residency or sovereignty commitment you have already made in writing, a contractual bar from a partner or a clinical site, a process network with no outbound path, a decision that must complete inside a control loop, or a validation obligation that a provider will not contract to support. When one of those is genuinely binding, a locally hosted model is a legitimate and sometimes the only answer. When none of them is, self-hosting is a large standing obligation taken on for a feeling.

The honest middle deserves more attention than it gets. Dedicated capacity, regional deployment, and a well-drafted schedule covering version pinning, deprecation notice, data use, processing locations, and support access will resolve most of what drives teams toward their own hardware, at a fraction of the burden. Where local inference is genuinely required, scope it properly: the hardware is the easy part, and a self-hosted model that nobody maintains is worse than the API it replaced. Decompose the task so a smaller model can succeed at anchored, constrained work, and keep the open-ended reasoning wherever the capability actually is.

Sakara Digital works with pharma and biotech organizations making these placement decisions and writing them down in a way that survives an inspection. If you are weighing a self-hosted deployment against a hosted one and want an independent view on which constraint is actually binding for your use case, we are happy to have that conversation.

For Further Reading