In short. SimpleQA evaluates short factual answers and separates correct responses, errors and abstention when calculating performance.
Definition: SimpleQA is a benchmark for evaluating short factual answers from language models. It distinguishes correct answers, incorrect answers and questions left unattempted, allowing factual accuracy and abstention to be measured together.
Key facts
- The paper Measuring short-form factuality in large language models introduces a dataset of 4,326 questions designed to have a single, verifiable answer.
- OpenAI's downloadable dataset contains
metadata,problemandanswercolumns. - As of 17 October 2026, OpenAI's reference evaluator uses a separate grading model to compare a response with its reference answer and assign a grade.
How are the questions constructed?
The SimpleQA paper describes human annotators writing questions with evidence-backed answers. Another annotator independently answers each question; agreement between the annotators is required for inclusion. Questions must specify the requested answer precisely, such as a city or a company.
The collection criteria require answers that remain stable over time and questions answerable by December 31, 2023. Collection also tests candidate questions against model responses and requires an incorrect response before retaining a question. This deliberately challenging selection matters when interpreting accuracy.
What counts as abstention?
Abstention means declining to supply the requested answer. The reference grading prompt uses these labels:
A: CORRECT
B: INCORRECT
C: NOT_ATTEMPTED
A correct response supplies the required reference information without a contradiction. An incorrect response contradicts the reference, including when the contradiction is expressed tentatively. A response receives the unattempted grade when it omits required information while avoiding a contradiction.
The same prompt accepts differences in capitalization, punctuation and ordering. Its examples also allow a partial answer accompanied by an explicit admission that the remaining information is unknown to receive the unattempted grade.
How do the scores treat unanswered questions?
The reference implementation reports the correct, incorrect and unattempted proportions separately. It also calculates correctness among attempted questions and an F-score, the harmonic mean of overall correctness and correctness among attempts.
With c correct answers, i incorrect answers and n unattempted questions, the metrics are:
Overall correctness = c / (c + i + n)
Correct given attempted = c / (c + i)
F-score = 2c / (2c + 2i + n)
The paper's Appendix B explains that guessing with a probability of correctness above 50% improves expected F-score compared with abstaining. Reading the separate error and abstention rates therefore preserves information about the model's answering behavior.
Related terms
FAQ
Does saying “I think” count as abstention?
The grading prompt permits hedging in a correct answer. A tentative contradiction receives an incorrect grade, so uncertainty language alone does not determine abstention.
What happens if every question is left unattempted?
The reference implementation assigns zero to correctness among attempts when there are no attempts. Its F-score calculation also returns zero when both component metrics are zero.
Does SimpleQA measure factual accuracy in long articles?
The paper's discussion identifies the relationship between short-answer factuality and factuality in lengthy responses as an open research question.
Sources
- OpenAI, “Measuring short-form factuality in large language models”, 7 November 2024
- OpenAI, “SimpleQA test set”, undated
- OpenAI, “simpleqa_eval.py”, accessed 17 October 2026
Check how a page reads to answer engines. Open the GEO score analyzer