Domain-Specific Language Models for Pharma: When They Beat General Models

What Counts as a Domain-Specific Language Model

Gartner’s definition is broad. In its October 2025 announcement of the top strategic technology trends for 2026, it described domain-specific language models (DSLMs) as “language models trained or fine-tuned on specialized data for a particular industry, function, or process.” The same release claims these models offer higher accuracy, reliability, and compliance for targeted business needs, and predicts that by 2028 over half of the generative AI models used by enterprises will be domain-specific.¹ Those are forecasts and positioning statements, not test results, and the rest of this article looks at what the test results say.

The broad definition hides real differences. A model pretrained from scratch on PubMed abstracts and a general frontier model with a pharma-themed system prompt can both be sold as “built for life sciences.” They behave differently, they fail differently, and they create different validation and supplier obligations. It helps to separate four kinds of thing that travel under the same label.

The distinction matters for two reasons. First, the evidence differs by type. Type 1 and narrow Type 3 models have a strong record on extraction tasks. Type 2 models have a weaker record than their launch papers suggest. Type 4 products should be judged on their content and retrieval, because the model underneath is usually a general one you could evaluate directly. Second, validation and supplier work differs by type. If your vendor fine-tuned the weights, you need to know on what data. If your vendor did not touch the weights, your questions are about the content library and the prompts.

Why the Idea Made Sense in the First Place

The case for domain pretraining was well founded when it was made. In 2020, Gururangan and colleagues showed across four domains, including biomedical papers, that a second phase of pretraining on in-domain text improved performance on classification tasks in both high- and low-resource settings, and that adapting to the task’s own unlabeled data helped further.² The same year, Gu and colleagues at Microsoft Research went further for biomedicine. They argued that in a domain with abundant unlabeled text, pretraining from scratch beats continuing to train a general model, and they released PubMedBERT and the BLURB benchmark, reporting new state-of-the-art results across a wide range of biomedical tasks.³

Those results were real, and the models they produced are still in wide use for extraction. What changed is the comparison. In 2020 the general models were the same size and vintage as the domain models. By 2023 the general models were far larger and trained on far more text, and the question became whether a much bigger general model already knew enough biomedicine that domain training added little.

What the Published Evidence Shows

The research record since 2023 is large and moves quickly, so it helps to group it by what each study tested. Read together, the studies are more consistent than the headlines suggest.

General Models Caught Up on Medical Question Answering

The first clear signal came from medical exam benchmarks. Google’s Med-PaLM 2, which combined an improved base model with medical domain fine-tuning and new prompting strategies, scored up to 86.5% on MedQA, the US medical licensing style question set, and physicians preferred its answers to those of other physicians on eight of nine clinical axes. It first appeared as a 2023 preprint and was published in Nature Medicine in 2025.⁴ That looked like a win for domain tuning.

In November 2023, Microsoft researchers tested whether a general model could match it without any domain training. Using GPT-4 with a set of general prompting methods they called Medprompt, they reported state-of-the-art results on all nine benchmark datasets in the MultiMedQA suite, a 27% reduction in error rate on MedQA compared with the best specialist methods to date, and a MedQA score above 90% for the first time. They stressed that the prompting methods were general purpose and made no specific use of domain expertise.⁵

Open domain models showed the same pattern. MEDITRON-70B, built by continuing to pretrain Llama 2 on curated medical text including PubMed and clinical guidelines, reported a 6% absolute gain over the best public baseline in its size class. Its own paper also reported that it was within 5% of GPT-4 and 10% of Med-PaLM 2.⁶ The domain model improved on its starting point and still trailed the larger general model.

“Medical” Versions of Open Models Rarely Beat Their Base Models

The most careful test of domain-adaptive pretraining came from Jeong and colleagues at EMNLP 2024. They compared seven public medical LLMs and two medical vision-language models head to head against the exact base models they were built from. They tuned prompts separately for each model and accounted for statistical uncertainty, practices they note are not consistently used in the literature. In the 3-shot setting, the medical models beat their base models in only 12.1% of cases, tied in 49.8%, and were significantly worse in 38.2%.⁷ Their conclusion was that strong general models may already have much of the medical knowledge and reasoning that domain pretraining was meant to add.

That result does not mean domain training never helps. It means the improvements claimed in many launch papers did not hold up once the comparison was fair. For a buyer, the practical lesson is about evaluation method: a vendor benchmark that compares its domain model with a different, older, or poorly prompted general model tells you little.

Fine-Tuned Small Models Still Win on Extraction and Classification

The picture reverses on structured tasks. Chen and colleagues, in a 2025 Nature Communications study, ran four LLMs from the GPT and LLaMA families across 12 biomedical NLP benchmarks in six application areas and compared them with the established approach of fine-tuning domain-specific BERT or BART models. The fine-tuned domain models had a macro-average of about 0.65 across the 12 benchmarks against 0.51 for the best zero- or few-shot LLM, and the gap was far larger on information extraction: 0.79 against 0.33 on relation extraction. On disease name recognition, the fine-tuned approach scored an F1 of 0.909, and GPT-4’s best zero- and one-shot result was about 30% lower.⁸

The same study found the opposite on reasoning. GPT-3.5 and GPT-4 outperformed the fine-tuned approaches on medical question answering and gave reasonable, though lower than state-of-the-art, results on summarization and simplification. The authors also reported a monetary trade-off: GPT-4 was 60 to 100 times more expensive than GPT-3.5 in their runs.⁸

Clinical text tells the same story. Lehman and colleagues tested 12 models ranging from 220 million to 175 billion parameters on three tasks that require parsing and reasoning over electronic health records. They found that relatively small specialized clinical models substantially outperformed all in-context learning approaches, even when fine-tuned on limited labeled data, and that pretraining on clinical text let smaller models match or beat much larger general ones.⁹

The Same Split Appears Outside Medicine

Finance offers a useful parallel because it produced the best-funded domain model of its time. BloombergGPT, a 50 billion parameter model, was trained on a 363 billion token financial dataset plus 345 billion tokens of general text, and its developers reported that it beat existing models on financial tasks by significant margins.¹⁰ Within months, Li and colleagues tested ChatGPT and GPT-4 on the same kinds of financial tasks. On aspect-based sentiment (FiQA), five-shot GPT-4 scored a weighted F1 of 88.11 against 75.07 for BloombergGPT. On commodity headline classification, GPT-4 scored 86.00 and BloombergGPT 82.20, but a fine-tuned BERT model beat both at 95.36.¹¹ The general model beat the expensive domain model, and the small fine-tuned task model beat them both on the narrow task.

Recent Frontier Models Against Specialized Clinical Tools

The most recent comparison we found tested products rather than research models. In a December 2025 preprint, Vishwanath and colleagues compared two widely deployed clinical AI assistants, OpenEvidence and UpToDate Expert AI, against GPT-5, Gemini 3 Pro, and Claude Sonnet 4.5 on a 1,000-item benchmark drawn from MedQA and HealthBench. The general models consistently outperformed the clinical tools, with GPT-5 scoring highest, and the clinical tools showed weaknesses in completeness, communication quality, context awareness, and safety reasoning.¹² These clinical tools are closer to Type 4 products than to domain-pretrained models, so the result speaks to “specialized product versus general model” rather than to domain pretraining itself. It is still a caution for anyone assuming that a tool marketed for a regulated field will outperform a current general model in that field.

Where Domain Models Still Win, and Where They Do Not

The studies point to one organizing idea: the shape of the task predicts the winner better than the domain does. Tasks with a fixed output, a fixed label set, and large volumes favor a small model trained for that task. Tasks that need reasoning, flexible language, or knowledge that changes often favor a strong general model, usually with retrieval.

Tasks Where a Trained Domain Model Has a Real Advantage

Extraction and classification against a fixed scheme are the clearest case. The adverse drug event work from Microsoft Research is a good example for pharma. Gu and colleagues used GPT-3.5 to label biomedical text and then distilled that knowledge into a PubMedBERT model. The distilled model was over 1,000 times smaller than the large models and outperformed GPT-3.5 by over 6 F1 points and GPT-4 by over 5 F1 points on standard adverse drug event extraction, without any human-labeled training data.¹³ The authors also pointed to advantages in efficiency and white-box model access.

Pharmacovigilance also shows how hard the full task is. BioDEX, a dataset built from 65,000 abstracts, 19,000 full-text papers, and 256,000 expert-created safety reports, asks a model to produce the core facts of a safety report from its source paper. The authors estimated human performance at 72.0% F1; their best model reached 62.3%.¹⁴ No model, domain or general, has solved document-level safety report extraction, which is a reason to keep people in that loop regardless of model choice.

In practice, the tasks that fit this pattern in pharma and biotech include:

• Coding adverse event verbatim text to a controlled terminology such as MedDRA, where the label set is fixed and volumes are high.
• Pulling defined fields (product, dose, event, onset, outcome) from case narratives or literature for pharmacovigilance intake.
• Classifying deviations, complaints, or change requests into an established taxonomy for trending.
• Screening literature abstracts for relevance against a fixed protocol.

For these, a small fine-tuned model can be more accurate, cheaper per item, faster, and more consistent than a general model, and its behavior is easier to pin down for validation.

Tasks Where a Current General Model Usually Wins

Open-ended reasoning, drafting, summarization, and question answering across varied material favor general models. MedHELM, a Stanford-led framework that tested nine frontier models on 35 benchmarks mapped to a taxonomy of 121 medical tasks developed with clinicians, found that the advanced reasoning models performed best (DeepSeek R1 with a 66% win rate and o3-mini with 64%), while Claude 3.5 Sonnet reached comparable results at about 40% lower estimated computational cost.¹⁵ The same study found that performance was strongest on clinical note generation and patient communication and weakest on clinical decision support and administration and workflow tasks, which is a reminder that “medical” is not one task.

In pharma terms, this covers drafting regulatory and CMC narrative sections, summarizing study reports, answering questions over a mix of SOPs and guidance, drafting medical information responses from approved content, and preparing first-pass investigation summaries. In each of these, the model must reason over the specific documents in front of it. The source of truth is your documents, not the model’s training data.

Domain Advantages Fade Quickly

One pattern runs through every comparison above: a domain model is built on a base model of a particular generation, and general models keep improving. MEDITRON improved on Llama 2 and still trailed GPT-4 when it was released.⁶ BloombergGPT was overtaken by a general model within months.¹¹ PharmaGPT’s developers reported that their 13 and 70 billion parameter models surpassed general models on domain benchmarks such as NAPLEX, the US pharmacist licensing exam, with a fraction of the parameters.¹⁶ That is the developers’ own evaluation, and like every such claim, it describes the general models available at the time it was written.

For a buyer, this means any domain model claim has a date attached. A two-year-old benchmark comparison against a two-year-old general model says almost nothing about whether the domain model beats the general model you could use today.

A Gap We Could Not Close From the Literature

The extraction studies above mostly used 2023-era general models. We did not find a published head-to-head comparison of 2026 frontier models against fine-tuned biomedical encoders on the same extraction benchmarks. It is reasonable to expect the gap has narrowed. It is not reasonable to assume it has closed. If an extraction task matters to you, test both options on your own data rather than relying on either the older studies or vendor claims.

Three Levers: Domain Model, Fine-Tuning, or Retrieval

When a general model falls short on a pharma task, the instinct is to look for a model that “knows pharma.” That is one of three different fixes, and each one addresses a different kind of failure. Choosing well starts with naming the failure.

Retrieval Fixes Missing or Changing Knowledge

If the model gives wrong answers because it does not know your SOPs, your product specifications, your current labeling, or this year’s guidance, the fix is to give it those documents at the time of the question. That is retrieval-augmented generation (RAG). Ovadia and colleagues compared unsupervised fine-tuning with RAG across a range of knowledge-heavy tasks and found that RAG consistently outperformed fine-tuning, both for knowledge the model had seen in training and for new knowledge. They also found that models struggle to learn new facts through unsupervised fine-tuning.¹⁷

Gekhman and colleagues found a sharper risk. In a controlled study, fine-tuning examples that introduced new facts were learned more slowly than examples consistent with what the model already knew, and as those new facts were eventually learned, the model’s tendency to hallucinate rose in proportion. Their conclusion was that models acquire factual knowledge mostly in pretraining, and fine-tuning mostly teaches them to use what they already know.¹⁸

For a regulated company this is decisive for most knowledge problems. Your controlled documents change through document control. A fact trained into weights cannot be superseded when the SOP is revised. A retrieved passage can, as long as the index follows the effective version. Retrieval also gives you a citation to check, which a fine-tuned model cannot.

Fine-Tuning Fixes Behavior, Format, and Consistency

If the model knows enough but does the task the wrong way (it will not stick to your taxonomy, it varies its output format, it is too slow or too expensive per item at your volume), fine-tuning is the better tool. Fine-tuning on a few thousand labeled examples teaches a model to produce one kind of output reliably. This is where the extraction results above come from.

Fine-tuning has also become far less demanding. Low-rank adaptation (LoRA) freezes the original model weights and trains a small set of added parameters. Its authors reported that, compared with full fine-tuning of GPT-3 175B, LoRA cut the number of trainable parameters by 10,000 times and GPU memory needs by three times, with model quality on par or better.¹⁹ That makes task-specific tuning practical for mid-size companies, and it produces a small adapter file that can be versioned and controlled separately from the base model.

Distillation is a related option worth knowing. A large general model labels examples, people check a sample, and a small model is trained on the result. The adverse drug event study above used this approach to get a small model that beat its teacher.¹³ It is often the fastest route to a high-volume extraction model when labeled data is scarce.

A Domain-Pretrained Model Fixes Unfamiliar Language

A domain-pretrained model helps most when the text itself is unlike anything general models have seen in volume: dense clinical shorthand, specialized notation, or domain vocabulary that general tokenizers split badly. The clinical results from Lehman and colleagues fit this case.⁹ Most pharma business text (SOPs, protocols, regulatory submissions, investigation reports) is formal English that general models handle well. Domain pretraining is the right lever less often than vendor marketing implies, and it is the most expensive lever to build and to keep current.

A Decision Sequence for Pharma Use Cases

The evidence supports a simple order of work. Start with the cheapest, most current option, measure it on your own data, and add specialization only where measurement shows a gap that specialization fixes. This sequence applies whatever the deployment arrangement. Where the model runs, and who controls its version, is a separate decision that our earlier article on localized LLMs covers.

How Common Use Cases Tend to Resolve

Applying the sequence to typical pharma and biotech work gives a fairly predictable pattern. The table below reflects our reading of the evidence, not a rule. Your evaluation set should overrule it whenever the two disagree.

Notice how rarely a domain-pretrained general assistant is the first choice. That is not a judgment on any particular product. It follows from the evidence: domain pretraining helps most on unfamiliar text and narrow tasks, and the narrow tasks are better served by a small model trained for exactly that task.

Validation Implications of a Domain-Tuned Model

Choosing a domain model or fine-tuning one changes what you have to validate and what evidence you need. Three points matter most for quality and computerized system validation (CSV) leaders.

Domain Training Does Not Change the Regulatory Category of a Generative Model

Draft Annex 22 to the EU GMP guidelines, published for consultation in July 2025, applies to AI models used in critical GMP applications with direct impact on patient safety, product quality, or data integrity. It covers static models with deterministic output and excludes generative models explicitly: “Following the above, the document does not apply to Generative AI and Large Language Models (LLM), and such models should not be used in critical GMP applications.”²⁰ For non-critical uses, the draft expects qualified people to remain responsible for confirming outputs are suitable, a human-in-the-loop arrangement.²⁰

Annex 22 is still a draft with no final text and no implementation date, and EMA has been gathering evidence on whether the generative AI position should change. Whatever the final text says, a medical or pharma LLM is still a large language model. Domain training changes what the model knows. It does not make its output deterministic. A vendor who implies that a “GMP-trained” LLM avoids the generative AI question has not read the draft.

If You Fine-Tune, Your Training Data Becomes Validation Evidence

FDA’s January 2025 draft guidance on using AI to support regulatory decision-making for drugs and biologics, which remains a draft, sets out a seven-step, risk-based credibility assessment framework. Model risk combines model influence (how much the AI output contributes relative to other evidence) and decision consequence (how serious an adverse outcome from a wrong decision would be).²¹ The draft asks sponsors to describe the data used to develop the model, meaning the training and tuning data, and says that data should be fit for use: relevant to the context, and reliable in the sense of accurate, complete, and traceable.²¹

The draft’s scope matters here. It does not address AI used in drug discovery or for operational efficiencies, such as internal workflows or drafting a submission, that do not affect patient safety, drug quality, or the reliability of study results.²¹ Many drafting and search uses fall outside it. Extraction and classification models used in pharmacovigilance, quality decisions, or study data handling are more likely to fall inside it.

The practical effect is that fine-tuning moves part of the validation burden onto you. With a general model you did not train, your evidence is about the system around it and its performance on your test set. Once you fine-tune, you also own the training set: where the examples came from, who labeled them, how disagreements were settled, whether they represent the cases the model will see, and whether any of them overlap with the evaluation set. Our earlier article on data quality checks for AI training sets covers that work in detail.

Change Control Has More Moving Parts

A domain-tuned system has at least three versioned components: the base model, the domain or task tuning (weights or an adapter), and the prompts and retrieval configuration around them. Each can change independently. A vendor can re-base its domain model on a newer general model, which is a significant change even if the product name stays the same. Your own team can retrain an adapter on new labeled data. Both need change control with re-testing against the same evaluation set.

The ISPE GAMP Guide: Artificial Intelligence, published in July 2025, frames these activities across the AI-enabled computerized system life cycle and defines responsibilities for both regulated companies and internal and external suppliers.²² It is the most practical reference for mapping who owns which component when a supplier provides the domain model and you provide the fine-tuning.

Supplier Implications: What to Ask a Vendor Selling a Pharma Model

As domain-specific language models for pharma become a product category, more vendors will claim domain training. The evidence above gives you specific questions that separate a real advantage from a label. These add to your normal supplier qualification work rather than replacing it.

Questions About the Model Itself

• What type is it? Pretrained from scratch, continued pretraining of a general model, fine-tuned, or a general model with domain retrieval. Many products are the last of these, and that changes every question that follows.
• What base model, and what version? If the vendor built on an open model, you can test the base model yourself. The Jeong results suggest you should.⁷
• What was it trained on, and when? Ask for the corpus at a level of detail you can assess: sources, dates, how it was filtered, and whether it includes regulatory text that has since been superseded.
• Against what was it compared? A comparison with an older or poorly prompted general model is weak evidence. Ask for results against a current general model on tasks similar to yours, with the prompts used.
• How often is it re-based or retrained, and how will you be told? This is the change control question, and it needs a contractual answer.

Questions About Your Data and the Weights

• Will our data be used to train or tune the vendor’s model? If yes, your documents become part of a product other customers use.
• If we fine-tune, who owns the resulting weights or adapter? Ownership decides whether you can move, retain, or retire the model on your terms, which matters for records retention and exit.
• Can we run our own evaluation set against each new version before it goes live for us? Without this, you cannot show the validated state held across a vendor update.

The EU AI Act Adds a Provider Question

Obligations for providers of general-purpose AI models under the EU AI Act entered into application on 2 August 2025, and the Commission’s enforcement powers started on 2 August 2026. The Commission’s guidelines on the scope of these obligations clarify that only those making significant modifications to a model need to comply with the obligations for providers of general-purpose AI models, not those making minor changes.²³ For most pharma companies doing ordinary task fine-tuning, that is reassuring. For a company or vendor doing large-scale continued pretraining to create a new domain model, it is a question for legal review: the party that substantially modifies a general-purpose model may take on provider duties for the result.

The same question is useful in the other direction. When a domain vendor builds on someone else’s general-purpose model, ask what technical documentation it holds from the upstream provider, what it can share with you, and whether its own modifications were large enough to make it a provider in its own right.

Watch the Product That Is a General Model Underneath

A Type 4 product, a general model behind a curated pharma content library, can be a good product. The value is in the content curation and the retrieval, and those can save months of work. But the qualification questions are different. You need to know which general model it uses and how it changes, how the content library is kept current, how citations are checked, and whether you can see which source passages produced an answer. The clinical tools study is a caution here: specialized products did not outperform current general models in that test, and their weaknesses showed up in completeness and safety reasoning, not in medical vocabulary.¹²

Do Not Overlook Open Domain Models

Open domain models are also improving. Google’s MedGemma collection, built on Gemma 3 at 4 and 27 billion parameters, reported clear gains over its base models on several medical tasks while keeping their general abilities, and further fine-tuning cut errors on an electronic health record retrieval task by 50%.²⁴ Open models of this kind give you something a closed domain product does not: the ability to test against the base model directly, fine-tune for your task, and hold a fixed version under your own change control. For narrow tasks where domain tuning does help, an open domain model plus your own task tuning can be a better position than a closed product whose training you cannot inspect.

Conclusion

The published evidence on domain-specific language models is clearer than the marketing around them. Small models trained for one extraction or classification task still outperform large general models on that task, and in GMP settings their fixed, repeatable behavior may matter even more than their accuracy. Domain-pretrained general assistants have a much weaker record: careful comparisons show they rarely beat the base models they came from, and current general models often beat them outright. Most knowledge problems in pharma are better solved by retrieval over controlled documents than by training facts into weights, because document control, not model training, is how a regulated company keeps knowledge current. The right question is not whether to buy a pharma model. It is which failure you are fixing, which lever fixes it, and what evidence from your own data shows that it worked.

Sakara Digital works with pharma and biotech organizations deciding where AI models fit in regulated work, including how to test domain claims, when fine-tuning is worth the validation effort, and what to ask suppliers before signing. If you are weighing a domain-specific model against a general one and want an independent view on how to test the choice, we are happy to have that conversation.

For Further Reading