In short. Retrieval supplies reference material, but RAGTruth records unsupported and contradictory claims even when models are instructed to use that evidence.
Definition: No. The RAGTruth study records unsupported and contradictory claims in retrieval-augmented generation (RAG), a method that supplies external reference material to a language model while it produces an answer.
Key facts
- The RAGTruth paper reports hallucinations in 43.1% of its 17,790 responses, across question answering, business descriptions and news summarization.
- Lewis and colleagues' RAG study reports more factual language than a comparison model without retrieval in its language generation experiments.
- Lost in the Middle finds that changing the position of relevant information affects performance in its tested question answering and key-value retrieval tasks, even when that information remains in the input.
What does RAGTruth actually test?
Published in August 2024, RAGTruth examines naturally generated responses from six models. Its question answering task supplies three retrieved passages per question and asks for answers based solely on those passages. The business description task supplies structured business data and customer reviews. The summarization task supplies news articles.
Human annotators mark the specific text spans containing unsupported or conflicting information. Each response receives independent annotations from two people, with a further review when their annotations differ considerably. The study therefore checks how generated statements relate to the material supplied to the model.
Its definition of a hallucination includes information absent from the reference even when that information might be true. The paper gives the example of claiming a restaurant accepts credit cards without supporting reference material. The reported percentage measures failures under this evidence-grounding standard in the sampled models and tasks; it supplies no current error rate for commercial answer engines.
What failures occur despite supplied evidence?
RAGTruth's Appendix A documents a restaurant example whose source contains the following field, shown here inside a JSON object:
{"RestaurantsReservations": false}
The generated description nevertheless says the restaurant takes reservations. This is a direct conflict with the supplied value. Another example supplies a true takeout field, while the response adds delivery as an available option. That additional service lacks support in the displayed source.
The study also observes that some models interpret a missing value, represented as null, as false. Its annotation rules count that conversion as unsupported information because the dataset uses an explicit false value or the text No for negation. Missing information and an explicit negative can therefore produce different judgments when checking a business description.
These examples give website operators concrete checks when reviewing an AI answer about a business: compare reservation claims with the supplied reservation value, check added services against the evidence, and preserve the distinction between missing and negative data.
How can evidence use be checked?
RAGTruth's annotation method compares reference material with individual response spans. A review following that approach keeps the supplied passages alongside the answer and marks the words that conflict with them or add unsupported information. Checking the retrieved material and checking the resulting claims are separate steps in that review.
The same paper tests an additional detector that selects between candidate responses. In the group using GPT-3.5-Turbo-0613 and GPT-4-0613, selecting responses with no detected hallucination spans leaves a measured hallucination rate of 4.8%. Detection improves selection in that experiment while residual errors remain.
Related terms
FAQ
Does an instruction to use only the supplied sources solve the problem?
No. RAGTruth's question answering prompt requires answers strictly based on the supplied passages and provides an abstention response when information is missing, yet the dataset contains unsupported answers.
Does unsupported always mean factually false?
No. RAGTruth explicitly labels information absent from the reference as a hallucination even when it may be true, and adds an annotation for such cases. If you audit an answer, identify whether the issue is a contradiction or an unsupported addition.
Can supplying more text ensure that the model uses the evidence?
Lost in the Middle reports weaker performance when relevant information appears in the middle of long inputs in its tested models. Its experiments distinguish the presence of evidence from successful use of that evidence.
Sources
- Association for Computational Linguistics, "RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models", August 2024
- arXiv, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", revised 12 April 2021
- MIT Press, "Lost in the Middle: How Language Models Use Long Contexts", 2024
Check how a page reads to answer engines. Open the GEO score analyzer