In short. FRAMES tests factual answer accuracy on 824 questions requiring retrieval and reasoning across multiple Wikipedia articles.
Definition: FRAMES (Factuality, Retrieval, And reasoning MEasurement Set) is a benchmark of 824 questions for evaluating systems that retrieve information and use it to produce answers. It tests factual answer accuracy on questions requiring information from multiple Wikipedia articles and reasoning across those sources.
Key facts
- The FRAMES paper, Fact, Fetch, and Reason introduces a unified evaluation of retrieval and reasoning in complete question-answering systems.
- Google's dataset card describes questions requiring information from 2 to 15 Wikipedia articles, with reference answers and relevant articles supplied.
- The published paper reports that human annotators create the questions and label them by reasoning type; a question can carry several labels.
What do the questions require?
Retrieval-augmented generation (RAG) means producing an answer with information retrieved from external documents. Multi-hop questions require combining information across sources to reach an answer. In FRAMES, finding relevant articles is part of the task, alongside using their contents correctly.
The dataset labels cover numerical reasoning, tabular reasoning, multiple constraints, temporal reasoning, and post-processing. Numerical reasoning includes calculations and comparisons. Tabular reasoning uses information in Wikipedia tables or infoboxes. Post-processing applies a requested transformation after gathering the facts, such as converting a resulting year into Roman numerals.
Annotators recheck whether answers are correct and supported by the associated Wikipedia pages. They also add time context to questions whose answers can change. These checks make the reference answer and its supporting articles explicit parts of the evaluation.
How does multi-step retrieval work?
The paper's retrieval pipeline uses BM25, a method for scoring documents against a query, over an index of Wikipedia articles. The model generates search queries from its current context. Retrieved articles are added to that context, with duplicates excluded, and the process repeats before the final answer.
The authors compare this pipeline with a direct question prompt, a single retrieval step, and an oracle prompt. Here, an oracle prompt supplies all the reference articles used by the annotators. For Gemini-Pro-1.5-0514, the paper reports answer accuracy of 0.408 without retrieval, 0.474 with four BM25-retrieved documents, and 0.729 with the oracle prompt.
With search-planning instructions, the multi-step system reaches 0.66 accuracy using five iterations, five queries per iteration, and ten documents per query. The instructions include examples of search sequences and directions to avoid repeated queries. These are results from the paper's experiments, published in 2025.
How is factuality scored?
A reference answer, also called a gold answer, is the expected answer supplied with a question. The paper's evaluation procedure uses a language model as an automatic judge because answers are free-form text. The judge checks whether the response contains the reference answer's meaning and essential facts, allowing equivalent wording.
Figure 6 includes this output format:
"Explanation:" (How you made the decision?)
"Decision:" ("TRUE" or "FALSE" )
The authors validate the automatic judge against human judgments and report agreement accuracy of 0.96. Answer accuracy is distinct from retrieval recall, which tracks how much of the required article set reaches the context. The paper reports both when analyzing the effect of retrieved evidence. The oracle experiment also shows that errors remain when all reference articles are supplied, including errors involving calculations and tables.
Related terms
FAQ
Does a FRAMES score measure citations to my website?
FRAMES evaluates answers to questions grounded in Wikipedia, using reference answers and article lists in the dataset. Measuring how often your website appears in an answer requires a separate evaluation.
How should I interpret a reported FRAMES result?
Compare the model, retrieval method, context supplied, and judge used. The paper reports different accuracies for direct prompting, BM25 retrieval, oracle context, and multi-step retrieval, and flags possible training-data contamination as a limitation.
Sources
- Association for Computational Linguistics, "Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation", April 2025
- Association for Computational Linguistics, "Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation", paper PDF, April 2025
- Google, "FRAMES: Factuality, Retrieval, And reasoning MEasurement Set", undated dataset card
Check how a page reads to answer engines. Open the GEO score analyzer