Lunit
TL;DR. We found a failure mode in clinical retrieval-augmented generation (RAG) that is invisible to every standard check. The model answers a question about drug X, but the documents it retrieved are about a different drug Y. It relays Y's real evidence, a real trial, a real outcome, and quietly staples it onto X. No hallucination fires. Faithfulness scores it clean. The citation is genuine. We call this deceptive grounding, and in a live deployment it shows up in 7.8% of answers, rising to 13.6% for recently approved drugs. Fine-tuning a model on medicine makes it worse.
Put yourself in a clinician's chair. You have a patient with a rare, brutal disease: fibrodysplasia ossificans progressiva, or FOP, where soft tissue slowly turns to bone. You are wondering about an old, well-understood antibody, rituximab, and whether anyone has tried it here. So you ask your decision-support tool the obvious question.
It answers in seconds. It is specific, it is confident, and it cites a real trial by its registry number. It looks exactly like the good answer you were hoping for.
Read it first. Then click the button underneath and let the system check its own work.
Rituximab has been studied in FOP. A 2023 phase-2 trial (NCT03188666) reported that rituximab reduced flare severity and duration, with meaningful improvement in quality of life.
Nothing here is a hallucination. The model did not make up a trial or a number. It read a document that really was in front of it and told you, accurately, what that document said. The only thing it got wrong is who the document was about. And that single mistake, the one thing none of our tools inspect, is enough to turn a correct-sounding answer into a clinically dangerous one.
We call this deceptive grounding. The response is grounded in retrieved evidence, faithfully so. The deception is in the attribution.
When we evaluate a RAG system for safety, we mostly ask three questions. Did it make things up? Is it faithful to its sources? Are its citations real? Those three checks are the backbone of RAG evaluation, and they are good checks. The problem is what they have in common: every one of them is satisfied when the model accurately relays a real document, which is exactly what deceptive grounding does. So all three pass, and they pass for the right reasons.
Looks for claims with no support. Every fact came from a retrieved document, so nothing fires.
Checks that the answer reflects the source. It does, faithfully. The entity mismatch is not a faithfulness question.
Confirms the cited document exists and is referenced correctly. NCT03188666 is real and correctly formatted.
Asks whether the evidence is about the drug you queried, and it is not. No standard framework runs this check.
It helps to line deceptive grounding up next to the failures we already know how to catch. A hallucination invents a fact, so a faithfulness or accuracy check can catch it. A knowledge conflict is when the retrieved document contradicts what the model believes, and that contradiction is a signal you can detect. Deceptive grounding has neither. It invents nothing, and it contradicts nothing. The retrieved evidence about Y lines up perfectly with what the model expected about X, because they are similar drugs for the same disease. It sits precisely in the blind spot: the case where the context and the model agree, and both are pointed at the wrong entity.
A RAG response can be simultaneously accurate, grounded, and clinically wrong. Accuracy and grounding are exactly the properties our safety checks measure, which is why they cannot see this.
We ran the same controlled stress test across 13 models, from small general-purpose models up to large ones fine-tuned specifically on medicine and biomedical text. If domain expertise were protective, the medical models should have been the safest; instead, they were the most dangerous.
The chart below is the peak deceptive-grounding rate for each model under the worst retrieval conditions we could construct. The two models fine-tuned on biomedical and medical corpora sit right at the top. A biomedical model misattributes in nearly nine out of ten answers at the peak.
Why would expertise hurt? Fine-tuning a model on medicine packs its weights with a cleaner, tighter map of pharmacological classes: which drugs belong together, share a mechanism, target the same pathway. That map is useful knowledge. But it is also exactly the structure that says "these two drugs are the same kind of thing," and once the model believes that, it becomes more willing to treat one drug's evidence as if it applies to its cousin. The stronger the class knowledge, the wider the door opens. This is not a knowledge gap. It is a knowledge strength, aimed at the wrong target.
An honest caveat: Those peak numbers come from a deliberately adversarial benchmark, engineered to trigger the failure as hard as possible. They measure susceptibility, not everyday prevalence. In a real deployment the rate is far lower. We will get to the real number at the end, and it is still high enough to matter.
Take the alternate drug Y and replace its real name with a completely made-up one, a drug that does not exist, while keeping the exact same clinical evidence attached to it. If the model were being tricked by a familiar name, a fake name should make things better; instead, it made things worse.
A drug that does not exist gets its evidence handed off more often than a real one. And when we replace a real name with an anonymous code like "XC-9941," misattribution climbs by 30 to 65 percentage points depending on the model. What drives the failure is the evidence: a trial, a number, an outcome that snaps neatly into the slot the model was already expecting for your drug. Content, not label.
If you are still holding onto the idea that this is a perception failure, that the model does not notice the two drugs are different, we tested that too. We labeled the alternate drug with a nonsense tag so obviously anomalous that the model would have to notice it.
The model flagged the mismatched entity in 80% of responses. Of the responses where it explicitly noticed, 73% still committed deceptive grounding. The misattribution happens downstream of perception, in the act of writing the answer.
Underneath all of this is a simple two-step structure, and you can watch it switch. Step one: shared disease context primes the model to treat Y's evidence as relevant to X, opening the door. Step two decides what walks through. If the retrieved document carries what we call completing information, the concrete specifics like trial names, registry numbers, and outcome figures, the model borrows those specifics and commits deceptive grounding. If that completing information is absent, the model has nothing to borrow, so it generates from memory instead, and you get plain confabulation.
The clean way to prove this is to change one thing and watch the outcome flip. Toggle the completing information below.
When we strip the specifics out, deceptive grounding goes to zero, and it looks like a win, until you notice the total error rate went up, to 98%. The model did not become correct. It stopped borrowing and started generating from memory.
For each claim in the answer, ask one more question that no standard framework asks: does the evidence supporting this claim actually belong to the drug I asked about, or to a different drug? We call it entity-attribution verification. It runs per claim, it needs no custom-trained model, and it bolts onto an existing evaluation pipeline.
Find the document behind each claim, read off which drug that document is really about, and flag the claim if that drug is not the one in the query. In our tests this catches deceptive grounding at 97% precision and 98.7% recall, and it fires on 0% of correctly-attributed answers.
Detection is necessary, but it is not where the biggest win is. If you can only do one thing, fix your retrieval. Every result in this paper points back to a single lever: the failure is triggered by the absence of the right document. When the system actually retrieves the queried drug's own evidence, deceptive grounding drops to at most 6.4%, from peaks in the 70s and 80s. Retrieving the right document is the highest-leverage intervention we found, by a wide margin.
And the tempting shortcut, just telling the model to be careful, only half works. An explicit instruction to verify the drug's identity before answering cut misattribution from 36% to 6% on the models driven by a strong internal prior. On other models it moved the number by four points and no more. Prompting is not a general fix, because, remember, the model already notices. The identity check has to be enforced before the model synthesizes its answer, not requested politely and ignored.
So the levers, in order of leverage: get retrieval right so the correct evidence is actually present; add entity-attribution verification to your evaluation so you can see the failure at all; and treat entity identity as a hard constraint in the model, not a suggestion.
We measured deceptive grounding in a real, deployed clinical decision-support system across 740 pre-registered drug and disease pairs, and it showed up in 7.8% of answers. For recently approved drugs, where the literature is thin and entity-specific evidence is hardest to retrieve, it rose to 13.6%.
New drugs are exactly the situation where a clinician is most likely to reach for an AI assistant, because they have the least experience to fall back on, and exactly the situation where the system is most likely to fill the gap with a similar drug's evidence dressed up under the new name. The failure concentrates precisely where the stakes are highest and the human is least equipped to catch it.
And it hides in plain sight, because it looks right. It is fluent, it is specific, it cites a real trial. Our existing safety nets, the ones that hunt for fabrication and unfaithfulness, will keep waving it through, not by accident but by design, because the failure lives below the level they inspect. A response can be accurate, grounded, and still about the wrong drug. Catching it means evaluating the one thing surface behavior conceals: not whether the evidence is real, but whether it is about the patient's drug at all.
@article{caruzzo2026deceptive,
title={Deceptive Grounding: Entity Attribution Failure in Clinical Retrieval-Augmented Generation},
author={Caruzzo, Cedric and Yoo, Donggeun and Kim, Tae Soo},
journal={arXiv preprint arXiv:2607.09349},
year={2026}
}