Duplicate Records in Pharmacovigilance: Detection Rules That Do Not Overtrigger

Why Duplicates Arise in Safety Databases

A duplicate, in EMA’s definition, is the same individual case reported by a primary source to describe suspected adverse reactions related to the administration of one or more medicinal products to an individual patient at a particular point in time.² The definition matters because it is narrower than “two records that look alike.” Two patients at the same clinic who received the same vaccine on the same day and both had an injection site reaction are not duplicates. They are two cases. A patient who told their physician about a rash, then filled in the company’s web form, then had the case picked up by a regional pharmacovigilance center, is one case with three records.

The routes that create duplicates are well understood. GVP Module VI Addendum I lists the common causes: a consumer and a healthcare professional reporting the same reaction; multiple healthcare professionals treating the same patient each reporting it; a reporter sending the same reaction to both the marketing authorization holder and a national competent authority; and literature reporting of the same reaction for generics, where every holder of a marketing authorization for the same active substance may create a case from the same article.² In practice there are several more routes that safety teams see every week:

• Partner exchange. Licensing and co-marketing agreements route cases between companies. Each company assigns its own case number, and if the receiving company’s intake process does not capture the sender’s identifiers, the case can be re-created rather than linked.
• Follow-up misfiled as new. A follow-up from a physician arrives by a different channel than the initial report (fax instead of phone, or a call center instead of the medical information line). If the intake team cannot find the original, they open a new case.
• Regulatory downloads re-imported. Marketing authorization holders download ICSRs from EudraVigilance for their products. In 2024, 220,734 download requests by marketing authorization holders returned 6,437,670 ICSRs from the database.⁷ A company’s own case, rerouted through a national authority and then downloaded, can come back looking like a new case if the import check does not recognize the identifiers.
• Literature reference variation. The same article cited three different ways (with and without the DOI, with the journal abbreviated differently, with an author name misspelled) defeats an exact-match check on the reference string. Hung and colleagues showed in 2023 that normalizing FAERS literature references to a standard citation format and applying fuzzy string matching raised the share of references that could be parsed from 61.74% to 93.93%.¹⁸

The consequence is not cosmetic. Duplicates distort the counts that signal detection depends on. The Addendum makes the point directly: since commonly used data-mining procedures may highlight associations with as few as three reports, one or two duplicates can severely affect their utility.² Hauben and colleagues, examining reports of quinine-associated thrombocytopenia in the FDA database, identified 20% of 141 reports as duplicates.¹⁷ A 2025 FDA study of 2,297 reports drawn from 12 pharmacovigilance reviews found that 901 (39%) were duplicates, forming 237 duplicate sets.¹³ Those are reports that reviewers had already pulled for evaluation, so the proportion is not a database-wide rate, but it shows how concentrated duplication can be exactly where the safety question is being asked.

That last figure is the one that should shape rule design. In the FDA sample, only 16% of confirmed duplicate sets agreed on all seven structured elements checked. Country of origin agreed 97% of the time, sex 94%, age 89%, suspect products 70%, concomitant products 62%, and the reaction preferred terms only 29%.¹³ A rule that requires the reaction terms to match will miss most true duplicates. A rule that requires nothing but country and sex to match will flag most of the database. The design problem lives between those two failures.

What the Regulators Expect

Three sets of documents define the expectation: EMA’s GVP Module VI and its Addendum I, the ICH E2B(R3) implementation guide that defines the identifiers every electronic case carries, and FDA’s regulations and regional technical specification. Read together, they say more about process and identifiers than about algorithms. None of them prescribes a threshold. All of them expect the organization to have a documented, validated, and audited way of finding and managing duplicates.

GVP Module VI Rev 2

Module VI is the EU’s operating guideline for collecting, managing, and submitting ICSRs. It became effective in its second revision on 22 November 2017, and the revision note states that it updated guidance on follow-up, duplicate detection, and data quality management to reflect the new EudraVigilance system.¹ Section VI.B.4 on data management requires that a procedure be in place to account for identification and management of duplicate cases at data entry and during the generation of aggregated reports.¹ That is two checkpoints, not one: when the case comes in, and again before it is counted in a periodic safety update report.

Section VI.C.6.2.4 sets the EU legal basis. Marketing authorization holders and Member State authorities are required to collaborate with EMA in the detection of duplicates under Articles 107(5) and 107a(3) of Directive 2001/83/EC, and to confirm that the quality system enables detection and management of duplicate ICSRs, they are to perform risk-based audits of the quality system at regular intervals, with corrective action and documented follow-up where deficiencies are found.¹ Appendix 7 of the module provides business process maps for the scenarios in which EMA’s own duplicate detection algorithm flags cases from the same sender or from different senders, and describes how EMA contacts the sender, how the sender confirms or disputes, and how the sender updates or nullifies cases in its own database.¹ Appendix 2.8 adds a specific expectation for literature cases: they should be checked against the organization’s database to identify articles that have already been submitted, including cases created by EMA’s own medical literature monitoring service.¹

GVP Module VI Addendum I

Addendum I (EMA/405655/2016) is the document that actually describes detection methods. It replaced the 2011 CHMP guideline on duplicate detection and became effective on the same day as Module VI Rev 2.² Its core positions, as a practitioner would summarize them:

• Every new case is a potential duplicate. As a general rule, every newly received ICSR should be considered a potential duplicate and checked thoroughly against the cases already in the database, at the time of data entry or when loading electronically received ICSRs.²
• Search on patient, reaction, and product similarity. Duplicate searches are generally based on similarities in patient, adverse reaction, and medicinal product data. For small systems a sorted table by age, sex, suspect products, and reactions may be enough; adding country can help. For E2B cases, screening the case identifier fields offers a quick start.²
• EudraVigilance groups first, then scores. In EudraVigilance, an initial grouping of ICSRs is performed on primary source country, sex, and age, and the algorithm then quantifies the difference between ICSRs statistically using additional patient, primary source, product, and reaction parameters.² This is the blocking-then-scoring pattern that most record linkage systems use.
• Handle missing values by widening, not excluding. If no match is found on the initial search, the search can be broadened to include null values, so a new female patient is checked against female patients and patients of unknown sex.²
• Match medical concepts, not exact terms. Differences in MedDRA coding should be handled by checking that the medical concepts are consistent. A search can start at Preferred Term level and move up to the Higher Level Term or Higher Level Group Term, bearing in mind that a rash can progress to Stevens-Johnson syndrome across reports.²
• Validate and tune the algorithm. The Addendum recommends carefully validating the duplicate detection algorithms of databases and evaluating the need to tune them over time, for example when data fields become mandatory.²
• Manual confirmation is always required. Upon identification of potential duplicates, a manual confirmation will always be necessary, and a well-documented case including a narrative is a prerequisite.² This is the sentence that makes overtriggering a compliance problem rather than an efficiency problem: every flagged pair is a manual review obligation.
• Keep records of every investigation. It is recommended to keep track of all duplicate investigations, including those where cases are confirmed not to be duplicates.²
• Do not let confirmation delay expedited reporting. If the individuality of cases cannot be confirmed without compromising legal expedited reporting timelines, enter the potential duplicate as a valid case and continue the investigation.²
• Root cause and PSMF. If duplicates are identified, analysis of the root cause should be performed and corrective action taken where appropriate, and the detection and management of duplicates must be addressed in the pharmacovigilance system master file.²

The Addendum also describes the two acceptable ways to manage a confirmed duplicate: allocate one existing case as the master and invalidate the others (they stay in the database for audit trail purposes but are not used for any other pharmacovigilance purpose), or create a new master case with a new worldwide unique identifier that combines the subordinates. EudraVigilance uses the second method. Either way, the master case must carry all the case reference numbers of the subordinates, the process must be well documented, and the original receipt dates of the duplicates must not change unless new information is received.²

ICH E2B(R3) identifiers

The E2B(R3) implementation guide (version 5.03, 18 July 2025) defines the fields that make duplicate detection tractable in the first place.³ Four are load-bearing:

Two more elements govern versions. C.1.2, the date of creation, is described in the guide as a timestamp that represents the equivalent of a version number: every iteration of an ICSR must carry a different value.³ C.1.5, the date of most recent information, must change each time follow-up information is received from a primary source, and must not change when a case is amended for internal reasons; in that situation C.1.11.1 is populated with the value for amendment.³ C.1.11.1 also carries the nullification flag, and the guide is explicit that an amended report keeps the same C.1.1 and C.1.8 previously submitted, while a case that must be resubmitted after nullification gets new identifiers.³

FDA: AEMS, 21 CFR 314.80, and the regional implementation guide

One current-status point first. FDA has renamed its adverse event reporting system. The pages that formerly described the FDA Adverse Event Reporting System (FAERS) now redirect to the FDA Adverse Event Monitoring System (AEMS), which FDA describes as consolidating multiple reporting systems across product categories, and the public dashboard is labeled as formerly FAERS.⁵ The published research on FDA duplicates still uses the FAERS name and this article follows the papers when citing them. What has not changed is the data limitation statement on the dashboard: duplicate and incomplete reports are in the system, there are many instances of duplicative reports, and some reports do not contain all the necessary information.⁵

The regulation itself, 21 CFR 314.80, is short on duplicate detection and clear on the obligations that create the risk. Applicants must submit each serious and unexpected adverse drug experience as a 15-day alert report, must promptly investigate, and must submit follow-up reports within 15 calendar days of receipt of new information; the regulation also states that ICSRs must only be submitted to FDA once, and requires that records of all adverse drug experiences be kept for 10 years.⁶ The FDA regional implementation guide for E2B(R3) (April 2024) tells sponsors to always use the same C.1.1 identifier assigned to the initial ICSR when submitting follow-up reports for the lifecycle of a case, and where an initial report went in on paper and the follow-up goes electronically, to carry the initial identifier in both C.1.1 and C.1.8.1.⁴ That is FDA’s version of the same principle EMA applies: the identifier is the primary defense, and the matching rules are the backstop for when the identifier was lost.

Follow-Up Version or True Duplicate

The most common source of false candidates in a company database is not two different reporters. It is the same case at two points in time. A follow-up is new information about a case the organization already holds, and it belongs in the same case as a new version. A duplicate is a second case record for the same patient-event-time occurrence. The two look alike to a similarity score: same patient, same drug, same event, overlapping dates. They are handled differently, and a rule set that cannot tell them apart will present follow-ups to reviewers as duplicate candidates over and over.

The identifiers settle it when they are present. If the incoming record carries the same C.1.8.1 as an existing case, and its C.1.5 date of most recent information is later, it is a follow-up. If it carries the same C.1.1 from the same sender, it is a follow-up or an amendment, depending on C.1.11.1. If it carries a C.1.9.1 identifier that matches an existing case’s C.1.1 or C.1.8.1, it is the same case arriving through another sender, which is exactly the situation the Addendum wants recorded rather than duplicated.

The hard cases are the ones without identifiers: a phone call from a nurse who says “I called about this patient last month,” a fax from a pharmacy with no reference number, a partner’s spreadsheet where the case column is blank. For these, the rule set needs a second decision after similarity: is this the same reporter adding to the same case, or a different reporter describing the same occurrence? Three signals help.

• Reporter identity. Same reporter name, institution, or contact details, with new clinical detail, points to follow-up. A different reporter type (consumer versus physician) points to a duplicate. In the FDA duplicate study, 89% of duplicates were reported by healthcare professionals versus 68% of non-duplicates, and 42% of duplicates mentioned another regulatory authority database versus 11% of non-duplicates.¹³ A narrative that mentions “also reported to” is a strong duplicate cue.
• Information direction. A follow-up typically adds outcome, dechallenge, lab values, or a corrected date to a record that already exists. A duplicate typically restates the initial event with a different level of detail and often a differently coded reaction. The Addendum’s caution that reactions can become more serious across reports (rash progressing to Stevens-Johnson syndrome) applies to both, which is why this signal is supporting evidence rather than decisive.²
• Receipt timing. A record arriving within the follow-up window after a query was sent to the reporter is most likely the answer to that query. Intake systems that track outstanding follow-up requests can use this to route the record to the existing case before any similarity scoring runs.

Designing the Matching Rules

Matching rules come in two families. Deterministic rules say “these fields agree exactly, therefore flag.” Probabilistic rules assign a weight to each field’s agreement or disagreement and sum the weights into a score, then compare the score to a threshold. Most working systems use both: deterministic rules on identifiers to catch the certain matches cheaply, and probabilistic scoring on demographic and clinical fields for everything else.

Deterministic keys

Deterministic keys should be reserved for fields where exact agreement really does mean the same case. In an E2B world that is the identifier set: C.1.8.1, C.1.1 from the same sender, and any identifier in C.1.9.1. A composite key that combines several non-identifier fields (country, sex, date of birth, suspect substance, reaction preferred term, onset date) is also deterministic in form, but it behaves like a very strict probabilistic rule: high precision, low recall. The FDA concordance figures show why. If only 29% of confirmed duplicate sets agree on preferred terms, a composite key that includes the preferred term will find fewer than a third of duplicates even before the other fields disagree.¹³

Literature references deserve their own deterministic key. A normalized reference (DOI or PubMed ID where available, otherwise first author, journal, year, and first page after normalization) combined with patient age and sex is a strong key for literature duplicates, and Hung and colleagues’ work on reference normalization shows how much recall depends on the normalization step.¹⁸

Probabilistic matching

The theory behind probabilistic record linkage is Fellegi and Sunter’s 1969 model, which frames the decision as three outcomes rather than two: link, non-link, and possible link, with the two thresholds set at stipulated error levels.¹⁹ The middle band is the one that goes to manual review. That three-way structure is the single most useful idea for controlling overtriggering, because it separates the pairs that are confident enough to auto-link (or auto-dismiss) from the pairs that need a person, and it lets the organization set the size of the review band deliberately.

The fields and their behavior, in rough order of evidential weight:

The vigiMatch method used by the Uppsala Monitoring Centre on VigiBase is a public example of this design. It compares report pairs on patient age, patient sex, onset date, a summary of all dates present in the report including those mentioned in the narrative, adverse events, and medicines or vaccines, plus an externally indicated flag that is set when a case identifier in one report matches the sender’s report identifier or the worldwide unique case identifier in another report.¹⁶ The original method was built on the hit-miss model for statistical record linkage, chosen because it handles limited training data well and suits categorical and numerical fields rather than free text.⁸ Its designers are careful to say it predicts suspected duplicates, not confirmed ones, and that both false positives and false negatives occur.¹⁶

Partial and missing fields

Missing fields are the norm, not the exception. Age is absent from many consumer reports, dates are often partial (month and year only), and reactions may be coded to a single vague term. The rule for missing data is simple to state and easy to get wrong in implementation: a missing field contributes no evidence in either direction. It is not an agreement and not a disagreement. Systems that treat null as a mismatch will never flag a sparse duplicate; systems that treat null as a match will flag every sparse case against every other sparse case. The Addendum’s instruction to widen a search to include null values is the right pattern: compare the female patient against females and unknowns, but weight the unknown comparison at zero.²

Partial dates need tolerance rules. If one record has an onset date of 2026-03-14 and the other has 2026-03, the pair agrees at month level and should receive the month-level weight. If one has 2026-03-14 and the other has 2026-03-21, a seven-day tolerance treats them as near-agreement, because reporters routinely give the consultation date rather than the onset date. The FDA 2022 error analysis found that mismatched data elements accounted for 290 of 331 errors examined, missing data for 62, and similar narratives across different patients for only 13, which is a useful reminder that most misses come from fields that disagree slightly rather than fields that are absent.¹¹

Blocking

Scoring every case against every other case is not feasible above a few tens of thousands of records. Blocking restricts comparisons to pairs that share a coarse key, which is what EudraVigilance does with its initial grouping on primary source country, sex, and age.² Good blocking keys are ones that duplicates almost always share (country agreed 97% of the time in the FDA sample) rather than ones that are merely convenient.¹³ Using two or three alternative blocking keys in parallel (country plus sex plus age band; suspect substance plus onset month; literature reference) catches pairs that one key would miss because of a single wrong value.

The Overtriggering Problem

Overtriggering is the failure mode in which the rule set flags far more candidate pairs than the team can review, and most of the flagged pairs are not duplicates. It is more common than undertriggering because it is invisible at design time. A threshold set low “to be safe” produces an impressive candidate count in the validation report and a queue that nobody can clear in production.

The workload arithmetic

A short illustration, using round numbers rather than any published rate. Suppose a mid-sized company processes 5,000 cases a month and its rule set flags 8% of incoming cases with an average of 1.5 candidate pairs each. That is 600 pairs a month. If a reviewer needs 15 minutes to open both cases, compare narratives, and document a decision (the Addendum requires that the decision be recorded either way), the queue needs 150 reviewer hours a month.² If the precision of the rule set is 20%, the team has spent 120 of those hours confirming that pairs are not duplicates. Raise the threshold so that the rule flags 3% of cases at a precision of 50%, and the queue drops to roughly 56 hours, of which 28 find real duplicates. The recall will be lower. Whether the trade is right depends on what the missed duplicates would have done to signal detection, which is the subject of prioritization below.

The point of the arithmetic is not the specific numbers. It is that every threshold decision is also a staffing decision, and a rule set designed without a staffing number attached is not designed.

Measuring precision and recall on a sample

Precision is the share of flagged pairs that are true duplicates. Recall is the share of true duplicates that were flagged. Neither can be read off the production system; both need labeled data. The published FDA and UMC work shows the range that real systems land in. Kreimeyer and colleagues’ 2017 probabilistic algorithm identified 77% of known duplicate pairs in VAERS at 95% precision, but only 13% of known duplicate pairs in FAERS at 100% precision, the difference reflecting how different the two systems’ data are.¹⁰ Their 2022 follow-on, which added narrative features, reported average pairwise recall of 0.71 and precision of 0.67 across 26 datasets, with wide standard deviations of 0.32 and 0.34.¹¹ The 2025 pipeline reported F1 scores between 0.36 and 0.93 across 12 expert-adjudicated datasets.¹² Tregunno and colleagues, evaluating vigiMatch on VigiBase reports from 2000 to 2010, found that 2.5% of evaluable reports were flagged as suspected duplicates (11% of literature reports, but only 0.5% of consumer reports), that the predictive value of a flag for a confirmed or likely duplicate ranged from 86% for the UK to 64% for Denmark and 33% for Spain, and that between 38% and 89% of the confirmed duplicates had not previously been found by the national centers’ own rule-based screening.⁹

Those spreads are the lesson. Performance varies by data source, by product type, and by dataset, so a vendor’s headline figure tells you very little about your database. The measurement has to be done on your data. A practical protocol:

Prioritizing candidate pairs by signal impact

Not every duplicate matters equally. A duplicate of a non-serious, listed event for a product with thousands of reports changes no count that anyone acts on. A duplicate of a serious, unlisted event for a product with four reports of that event may be the difference between a signal and no signal, which is the Addendum’s point about data-mining procedures that highlight associations with as few as three reports.² The queue should be ordered by impact, not by score alone or by arrival time.

A workable impact score multiplies three factors: seriousness (both cases serious ranks above one or none), rarity of the drug-event combination in the database (fewer existing reports means each duplicate is a larger share of the count), and regulatory context (the combination is in an active signal evaluation, the product is under additional monitoring, or a PSUR data lock point is within the next 60 days). Pairs where both cases are serious and the combination has fewer than ten reports go to the top of the queue regardless of score band. The FDA study found that 97% of duplicates in its sample had serious outcomes versus 83% of non-duplicates, which is consistent with duplicates concentrating exactly where reviewers are looking.¹³

A Worked Rule Set with Sample Thresholds

What follows is an illustrative rule set of the kind a safety team might document. The weights and thresholds are examples for a mid-sized post-marketing database, not published values, and any organization adopting a structure like this must calibrate the numbers on its own labeled data using the protocol above. The structure has three tiers.

Tier 0: deterministic identifier rules

• R0.1 Incoming C.1.8.1 equals an existing case’s C.1.8.1. Outcome: same case. If incoming C.1.5 is later than the existing case’s C.1.5, process as follow-up. If C.1.11.1 indicates amendment or nullification, process accordingly. No reviewer decision required beyond the normal case processing.
• R0.2 Incoming C.1.9.1.r.2 (any value) equals an existing case’s C.1.1 or C.1.8.1, or an existing case’s C.1.9.1.r.2. Outcome: same case received through another sender. Route to the reconciliation queue at top priority; the reviewer confirms and records the additional identifier on the master case.
• R0.3 Incoming record’s EudraVigilance local report number (for downloaded ICSRs) matches a case the company has previously submitted or downloaded. Outcome: do not create a new case; link the download to the existing case.
• R0.4 Literature case: normalized reference key (DOI or PubMed ID; otherwise first author surname, journal, year, first page) equals an existing literature case’s key, and patient sex agrees or is unknown in one, and age agrees within two years or is unknown in one. Outcome: high-priority review band.

Tier 1: blocking

Candidate pairs are generated only within blocks. Three blocking keys run in parallel and their outputs are unioned: (a) primary source country plus patient sex plus five-year age band, with unknown sex or age matching all values; (b) suspect active substance plus onset month, with a one-month tolerance on either side; (c) normalized literature reference key. A pair generated by any block proceeds to scoring.

Tier 2: probabilistic score

Decision bands

• Score 26 and above: high-priority review band. Expected precision in calibration: above 70%. Reviewer decision required within the case processing timeline.
• Score 18 to 25: standard review band, ordered by the impact score described above. Expected precision in calibration: 30% to 70%.
• Score 12 to 17: deferred band. Not reviewed routinely. Re-scored when either case receives follow-up, and reviewed in full before any signal evaluation or PSUR that includes the drug-event combination.
• Score below 12: auto-dismissed. Logged, not reviewed, unless a Tier 0 rule also fired.

The deferred band is what keeps the queue clearable without losing the pairs entirely. The Addendum recognizes that duplicate review is also necessary when evaluating signals, in addition to routine screening, and the deferred band is a way to honor that without reviewing every marginal pair as it arrives.² The organization must be able to show an inspector that deferred pairs are actually reviewed at the trigger points, which is a matter of the system producing a report of deferred pairs by drug-event combination on demand.

Upstream Fixes That Reduce Duplicates at Intake

The best duplicate is the one that never gets created. Most of the routes described earlier can be closed or narrowed at intake, before the matching rules ever see the record. These are data quality controls in the ordinary sense, and they belong in the intake SOP rather than the duplicate detection SOP. Sakara Digital’s earlier article on case intake failure points covers intake data quality in general; the items below are the ones that specifically reduce duplication.

Each of these has a measurable effect on the matching rules. Better identifier capture moves pairs from the probabilistic tier to the deterministic tier, where precision is near certain. Consistent substance coding removes a whole category of false disagreement. A follow-up register removes the largest single source of false candidates. Teams that measure candidate volume by intake route before and after these changes usually find that the intake work does more to shrink the queue than any amount of threshold tuning.

Machine Learning Approaches: What the Published Work Shows

Duplicate detection is one of the few pharmacovigilance tasks where machine learning has a long, public, peer-reviewed track record at regulators, which makes it a more grounded subject than most AI conversations in safety. This section is deliberately short. Sakara Digital has covered the broader regulatory position on AI in signal detection and the intake data problems that determine whether any model can work in the case intake article; neither is repeated here.

The published lineage runs from statistical record linkage to supervised classifiers. Norén and colleagues published the hit-miss model approach for the WHO database in 2007, designed for anonymized categorical and numerical data with limited training material.⁸ Tregunno and colleagues evaluated that method as vigiMatch on VigiBase in 2014, matching on reported drugs, reactions, patient age, sex, country of origin, and onset date, and reported the predictive values cited above, along with the observation that the probabilistic method found duplicates that rule-based methods had missed while producing fewer suspected duplicates to review.⁹ At FDA, Kreimeyer and colleagues combined probabilistic linkage on structured fields with clinical and temporal features extracted from narratives by natural language processing, first in 2017, then in a 2022 deployment where reviewers gave feedback across 26 datasets, and then in a 2025 pipeline that added network-based community detection to group duplicates into sets rather than pairs.¹⁰¹¹¹² The 2025 paper reports that the pipeline identified approximately 5 million duplicate reports among 29 million FAERS reports and is operating at FDA to provide deduplicated input for safety review operations.¹² Most recently, Barrett, Erlanson, Félix China, and Norén described a successor to vigiMatch that replaces the probabilistic model with separate support vector machine classifiers for medicines and for vaccines, adds country-specific features, and extracts temporal information from free text; they report precision of 92% for vaccines and 54% for medicines against 41% for the comparator method, with recall of 80% to 85% for vaccines and 40% to 86% for medicines against 24% to 53% for the comparator.¹⁴

Three lessons from that body of work apply directly to a sponsor’s rule set.

• Narratives carry the signal that structured fields lose. The FDA duplicate characterization study found only 29% agreement on preferred terms among confirmed duplicates but a median narrative cosine similarity of 0.87.¹³ Even a simple narrative similarity feature, of the kind included in the worked rule set, recovers duplicates that coding differences hide. Full natural language processing of dates and clinical events from narratives is what moved the FDA and UMC systems forward.
• Labels are the constraint. Every published system was trained and evaluated on expert-adjudicated pairs, and the performance spread across datasets (F1 from 0.36 to 0.93 in the 2025 FDA paper) shows how much depends on the labeled data matching the production data.¹² A sponsor that has not kept records of past duplicate investigations, as the Addendum recommends, has no training set and no validation set.²
• Grouping matters as much as pairing. Duplicates come in clusters, not just pairs, and the Addendum uses the term duplicate cluster for exactly this reason.² The UMC approach applies complete-link clustering after pairwise prediction and picks the master report by the highest completeness score; the FDA 2025 pipeline uses network community detection.¹⁶¹² A rule set that stops at pairs will present the same cluster to reviewers three times.

The governance position is set by the CIOMS Working Group XIV report on artificial intelligence in pharmacovigilance, published in December 2025, which covers all aspects of ICSR processing and warns that automation efforts focused only on efficiency risk removing the value added by humans in the loop.²⁰ For duplicate detection specifically, the practical reading is that a model may rank and prioritize candidates, and may safely auto-dismiss pairs below a validated floor, but that confirmation stays with a person, which is also what the Addendum requires.² A 2024 scoping review of 58 publications on duplication in pharmacovigilance databases concluded that efficient prevention, detection, and management are essential for reliable data and recommended worldwide unique identifiers alongside advances in artificial intelligence, which is a fair summary of where the field stands: identifiers first, models second, people for the decision.¹⁵

Documenting the Rule Set for Inspection

An inspector who asks about duplicate management is usually testing three things: whether the organization can describe its method, whether it can prove the method works, and whether it can show the method is used. A rule set that exists only as vendor configuration fails the first test. A rule set that was validated once at go-live and never re-measured fails the second. A queue with 4,000 unreviewed candidates fails the third, and that is the failure that overtriggering produces.

The documentation set that answers all three:

1. Rule specification. A controlled document listing every deterministic rule, every blocking key, every scored comparison with its weights and tolerances, and the decision bands with their thresholds, in the form shown in the worked example. Written so that a reviewer can recompute a score by hand for any pair.
2. Validation report. The labeled reference set (how it was built, how many pairs, from which sources), the recall by case type, the precision by score band from the production sample, the reviewer agreement statistics, and the staffing assumption behind the review threshold. The Addendum’s recommendation to validate the algorithms carefully is the basis for this document.²
3. Change control log. Every threshold or weight change, with the re-measurement that justified it. Changing a threshold changes which cases are counted in aggregate reports, so it is a validated system change and should be handled as one.
4. SOP and work instruction. The intake-time check, the periodic screening cadence, the pre-aggregate-report screening required by Module VI section VI.B.4, the review procedure, the master case allocation or creation procedure, the nullification procedure, and the process for notifying EMA of suspected duplicates in EudraVigilance.¹²
5. Investigation register. A record of every candidate pair reviewed, the decision, the reviewer, the date, and the reason, including pairs confirmed not to be duplicates, which the Addendum specifically recommends keeping.² This register is also the training data for the next validation cycle.
6. Queue metrics. Candidate volume by month and by intake route, confirmation rate by band, median and maximum age of open candidates, and deferred-band review completion before each signal evaluation and PSUR. These are the numbers that show the method is used and that it is not overtriggering.
7. Root cause and CAPA records. For confirmed duplicates, the intake route that produced them and what was changed, which the Addendum expects and which feeds back into the upstream fixes.²
8. PSMF section and audit evidence. The pharmacovigilance system master file must address duplicate detection and management, and Module VI expects risk-based audits of the quality system that confirm the detection process works, with documented dates and results.¹²

Conclusion

Duplicate detection in pharmacovigilance is a solved problem in the sense that the methods are known, published, and in daily use at EMA, FDA, and the Uppsala Monitoring Centre. It is an unsolved problem in most sponsor organizations because the rule set was configured rather than designed, its precision was never measured on the organization’s own data, and the review queue that resulted was sized by accident. The regulatory expectation is clear even though no regulator prescribes a threshold: every new case is a potential duplicate, every candidate needs manual confirmation, every investigation is recorded, the method is validated and tuned over time, and the identifiers defined in E2B(R3) are captured and carried faithfully so that the matching rules are a backstop rather than the primary defense. Overtriggering is what happens when the backstop is asked to do the whole job.

Sakara Digital works with pharma and biotech organizations that need their safety data to hold up: to signal detection, to aggregate reporting, and to inspection. If you are looking at a duplicate candidate queue that never clears, or preparing to validate a matching rule set and want an independent view on where the thresholds should sit and how to prove it, we are happy to have that conversation.

For Further Reading