In short. FreshQA evaluates factual question answering with changing reference answers and separate checks for main-answer correctness and factual detail.
Definition: FreshQA is a factual question-answering benchmark introduced in the FreshLLMs paper. It evaluates whether language models answer questions requiring current world knowledge and identify questions with factually incorrect premises.
Key facts
- The published paper describes 600 questions, divided into a test set of 500 and a development set of 100.
- The paper's abstract presents a two-mode human evaluation of correctness and hallucination.
- As of 17 October 2026, the authors' repository links a dataset labelled FreshQA April 21, 2026, alongside older versions.
How does FreshQA represent changing facts?
The paper's question taxonomy distinguishes never-changing answers, slow-changing answers that typically change over several years, and fast-changing answers that typically change within a year or less. It also includes false-premise questions, whose factual assumptions require rebuttal.
The dataset CSV records false_premise separately from fact_type. This selected-column extract shows how a false-premise question also carries a temporal classification:
question,false_premise,num_hops,fact_type
What year did the first human land on Mars?,TRUE,one-hop,slow-changing
Other fields include effective_year, next_review and supporting source URLs. The answer columns allow multiple accepted answers. These fields make the reference answer and its review schedule inspectable alongside the question.
How are responses evaluated?
The paper's evaluation procedure defines Relaxed as assessing the main answer and Strict as additionally checking every fact for accuracy. Both require a correct, clearly stated or inferable primary answer; extra information must avoid contradicting it. False-premise questions receive credit only when the response identifies the faulty premise.
Consequently, a correct main answer accompanied by an inaccurate supporting detail can pass Relaxed and fail Strict. A reported accuracy therefore needs its evaluation mode to be interpretable.
How can an evaluation be reproduced?
The authors' FreshEval instructions describe an automatic evaluator that uses example judgments to teach a language model to rate responses. The workflow copies the dataset spreadsheet, adds columns for model responses, ratings and explanations, and runs the notebook for the selected evaluation mode. It then uses true-or-false ratings to calculate accuracy by question category.
The same instructions distinguish FreshEval from answer-overlap measures such as exact match or recall. A comparison should identify the dated dataset, evaluation mode and evaluator or metric. Keeping the response and explanation columns also leaves a record for reviewing individual judgments.
Related terms
FAQ
Can I evaluate answers automatically?
Yes. The authors provide FreshEval notebooks for Relaxed and Strict evaluation, together with sample evaluation spreadsheets and results. Record which mode and judging model you use.
Why should I record the dataset version?
The repository lists dated datasets and invites corrections to questions and answers. Recording the version identifies which reference answers your evaluation uses.
Does a correct name always earn credit?
The paper's scoring rules also consider contradictory information and false premises. Under Strict evaluation, inaccurate additional facts can make a response incorrect.
Sources
- Association for Computational Linguistics, "FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation", August 2024
- Association for Computational Linguistics, "FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation", full paper, August 2024
- FreshLLMs authors, "FreshLLMs" repository README, accessed 17 October 2026
- FreshLLMs authors, "FreshQA April 21, 2026", dataset CSV
Check how a page reads to answer engines. Open the GEO score analyzer