RAGTruth corpus

Definition: RAGTruth is a human-annotated dataset for detecting claims that contradict, or lack support in, the context supplied to a language model. Its responses use retrieval-augmented generation (RAG), a method that supplies retrieved information to a model when it generates text.

Key facts

• The August 2024 RAGTruth paper describes nearly 18,000 naturally generated responses with manual annotations at response and word levels.
• The full paper covers question answering, data-to-text writing and news summarization, with six model responses per input.
• As of October 2026, the authors' repository documentation describes separate response and source files, connected through source_id, and labels each response's split as train or test.

What evidence does the corpus contain?

The paper's construction section describes the evidence supplied for each task. Question answering uses daily-life questions sampled from MS MARCO, with three retrieved passages per question. Models receive instructions to answer solely from those passages.

Data-to-text writing means producing prose from structured records. RAGTruth uses restaurant and nightlife business records from the Yelp Open Dataset, supplemented with up to three customer reviews. This context appears in JSON format. News summarization uses articles from CNN/Daily Mail and a separate collection of recent news.

These inputs make the grounding check concrete: an answer can be compared with its retrieved passages, and a business overview can be compared with the supplied business attributes and reviews. The unit of evidence is the context attached to the response.

How are unsupported spans annotated?

A span is a continuous portion of response text. The paper's annotation method assigns spans to four categories: evident conflict, subtle conflict, evident introduction of baseless information, and subtle introduction of baseless information. Conflict concerns disagreement with the context; baseless information adds material without support in that context. Each response receives independent labels from two annotators, with a third review for substantial disagreement.

The same paper distinguishes evidence support from whether a statement could be true outside the supplied context. Its implicit_true annotation marks information absent from the reference that may nevertheless be truthful. Its due_to_null annotation marks errors caused by interpreting a missing value as false in business records. This matters when a field leaves a business attribute unspecified.

The repository's file descriptions give the following compact field map:

response.jsonl
  source_id
  response
  labels: start, end, text, label_type

source_info.jsonl
  source_id
  task_type
  source_info
  prompt

start and end locate an annotated span in the response; text contains the span itself. Matching source_id connects the generated response to its context and original prompt.

What can a RAGTruth evaluation measure?

The paper's evaluation section distinguishes response-level detection, which identifies whether a response contains a hallucination, from span-level detection, which locates the affected text. Here, hallucination means content that conflicts with or lacks support in the supplied evidence.

The authors report precision, recall and F1 for response-level detection. Precision measures how many flagged responses actually contain hallucinations; recall measures how many affected responses are found; F1 combines both. Span-level evaluation measures overlap with human annotations at the character level.

For an agency assessing a tool that checks answer support, this distinction separates finding an affected answer from locating the exact unsupported wording. The authors identify generalization across other datasets and contexts as an area requiring further evaluation.

Related terms

• Retrieval-augmented generation (RAG)
• Grounding
• Hallucination
• Answer attribution

FAQ

Can a true statement receive a hallucination label?

Yes. Under the paper's strict grounding criteria, information absent from the supplied reference can receive a hallucination label even when it may be true. The implicit_true flag records this case.

How can I connect a response to its evidence?

Match source_id in response.jsonl to the same field in source_info.jsonl, as described in the repository README. The source record contains the context and prompt; the response record contains the generated text and span labels.

Sources

1. Association for Computational Linguistics, "RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models", August 2024
2. Association for Computational Linguistics, "RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models" (full paper), August 2024
3. ParticleMedia, "RAGTruth repository README", undated

---

Check how a page reads to answer engines. Open the GEO score analyzer