AI models often give the right answers but point to the wrong sources

Background

The central issue is not ordinary factual error, but a more subtle mismatch between answer correctness and evidentiary correctness. The article highlights that leading models can analyze documents and arrive at the right conclusion, yet cite passages that do not support that conclusion. This creates a false impression of transparency: the model appears to show its work, but the evidence trail is unreliable.

Peking University researchers label this behavior attribution hallucination. The term matters because it distinguishes citation failure from standard hallucination. A model may not invent the final answer; instead, it invents or misassigns the justification.

Key Points

• Correct answers are not enough. The article’s main claim is that model evaluation often overvalues answer accuracy while underexamining whether the cited text actually backs the answer.
• Citation errors are systematic enough to benchmark. The researchers created CiteVQA specifically to test this problem, indicating that attribution hallucination is not an edge case but a recurring pattern worth measuring directly.
• The risk is highest in regulated domains. In law and medicine, a wrong citation can be as dangerous as a wrong answer because decisions depend on traceable support, not just plausible output.

Why the Problem Is Serious

The article implies a breakdown in a basic assumption of explainable AI systems: if a model cites a passage, users tend to trust that passage as valid support. But when citations are disconnected from the answer, verification becomes misleading rather than helpful.

This is especially concerning in fields where documentation is part of compliance and accountability:

• In law, a conclusion without the correct supporting text can distort interpretation or precedent.
• In medicine, a recommendation tied to the wrong evidence can undermine clinical trust and potentially affect care decisions.

So the harm is twofold:

1. Users may accept unsupported reasoning because the answer looks documented.
2. Reviewers may waste effort checking irrelevant passages while missing the true basis—or absence—of support.

What CiteVQA Changes

The significance of CiteVQA is that it shifts evaluation from “Did the model answer correctly?” to “Did the model answer correctly and ground that answer in the right text?” That is a more demanding and realistic standard for document-based AI.

This is important because document QA systems are often marketed as auditable precisely because they quote sources. The benchmark challenges that assumption by testing whether the sourcing mechanism is dependable, not just present.

Broader Significance

The article’s strongest insight is that citation can create a veneer of reliability. A model that is right for the wrong reasons may still appear trustworthy, especially to non-expert users. That makes attribution hallucination a governance problem, not just a technical flaw.

By identifying and benchmarking this failure mode, the researchers push evaluation toward a more rigorous notion of trustworthiness:

• Answer accuracy
• Evidence relevance
• Faithful attribution

The broader implication is that future AI systems used for serious document work cannot be judged solely by output quality. They must also demonstrate that their cited evidence genuinely supports their claims. In that sense, CiteVQA addresses a gap between apparent explainability and actual verifiability.