Lunit
TL;DR. A clinical retrieval-augmented generation (RAG) system can answer a question about drug X with real evidence about drug Y. The facts and citation are genuine, so standard hallucination and faithfulness checks pass. We call this deceptive grounding. It appeared in 7.8% of answers from a live deployment and 13.6% for recently approved drugs. Medical fine-tuning made some models more susceptible.
Suppose a clinician asks whether rituximab has been used to treat fibrodysplasia ossificans progressiva (FOP), a rare disease in which soft tissue turns to bone.
The system gives a specific, confident answer and cites a real trial by registry number.
Read the answer, then use the button to verify it.
Rituximab has been studied in FOP. A 2023 phase-2 trial (NCT03188666) reported that rituximab reduced flare severity and duration, with meaningful improvement in quality of life.
The model did not invent the trial, registry number, or outcome. It accurately summarized a retrieved document but assigned that evidence to the wrong drug. Standard checks do not inspect that attribution.
We call this deceptive grounding: the response is faithful to its source but attributes the source to the wrong entity.
RAG evaluations usually ask whether the model invented a claim, whether the answer follows its sources, and whether its citations are real. Deceptive grounding passes all three because it accurately relays a real document. The missing check is whether that document concerns the entity in the query.
Looks for claims with no support. Every fact came from a retrieved document, so nothing fires.
Checks that the answer reflects the source. It does, faithfully. The entity mismatch is not a faithfulness question.
Confirms the cited document exists and is referenced correctly. NCT03188666 is real and correctly formatted.
Asks whether the evidence is about the drug you queried, and it is not. No standard framework runs this check.
A hallucination invents a fact. A knowledge conflict occurs when retrieved evidence contradicts the model's prior. Deceptive grounding does neither. Evidence about Y can match what the model expects about X because the drugs are similar and target the same disease. Context and prior agree, but both point to the wrong entity.
A RAG response can be accurate about its source and still be clinically wrong about the queried drug.
We ran the same controlled stress test across 13 general and medically fine-tuned models. The medical models were not consistently safer. Two had the highest peak failure rates.
The chart reports each model's peak rate under adversarial retrieval conditions. OpenBioLLM-70B misattributed evidence in 86.7% of answers at its peak.
One explanation is that medical fine-tuning strengthens knowledge of pharmacological classes: which drugs share a mechanism or pathway. That knowledge is useful, but it can also make evidence for one drug seem applicable to a related drug. The failure may come from overgeneralizing domain knowledge rather than lacking it.
Caveat: These peaks come from an adversarial benchmark designed to trigger the failure. They measure susceptibility, not normal prevalence. Deployment rates appear below.
We replaced drug Y's name with a fictional one while keeping its clinical evidence unchanged. If name familiarity caused the error, this should have reduced it. Instead, misattribution increased.
Replacing a real name with a code such as “XC-9941” raised misattribution by 30 to 65 percentage points, depending on the model. The trial details and outcomes drive the failure more than the drug label.
We also tested whether the model simply failed to notice the mismatch by giving the alternate drug an obvious nonsense label.
Models flagged the mismatch in 80% of responses. Among those responses, 73% still attributed the evidence incorrectly. Detection alone did not prevent the error during answer synthesis.
Shared disease context first makes evidence about Y seem relevant to X. What happens next depends on whether the document contains completing information, such as trial names, registry numbers, and outcomes. With those details, the model borrows them and misattributes the evidence. Without them, it generates from memory and confabulates.
The control below shows the effect of removing only the completing information.
Removing the specifics reduced deceptive grounding to zero, but raised the total error rate to 98%. The model stopped borrowing and started generating unsupported claims from memory.
We add one check per claim: does the supporting evidence concern the drug in the query? This entity-attribution verification requires no custom-trained model and can be added to an existing evaluation pipeline.
The verifier identifies the entity in each supporting document and compares it with the query. In our tests, it reached 97% precision and 98.7% recall, with no flags on correctly attributed answers.
Better retrieval had the largest effect. When the system retrieved evidence about the queried drug, deceptive grounding fell from peaks in the 70s and 80s to at most 6.4%.
Prompting was inconsistent. An instruction to verify drug identity reduced misattribution from 36% to 6% for models driven by a strong prior, but improved other models by no more than four points. The system needs to enforce identity before answer synthesis rather than rely on a reminder.
In practice: retrieve entity-specific evidence, add entity-attribution verification, and enforce entity identity as a constraint during synthesis.
Across 740 preregistered drug-disease pairs in a deployed clinical decision-support system, deceptive grounding appeared in 7.8% of answers. It rose to 13.6% for recently approved drugs, where entity-specific literature is sparse.
Clinicians are also more likely to seek help with new drugs because they have less experience with them. That makes the higher failure rate especially concerning.
These answers look reliable because they are fluent, specific, and supported by a real citation. Catching them requires checking not only whether the evidence is real, but whether it is about the drug the clinician asked about.
@article{caruzzo2026deceptive,
title={Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation},
author={Caruzzo, Cedric and Yoo, Donggeun and Kim, Tae Soo},
journal={arXiv preprint arXiv:2607.09349},
year={2026}
}