In short. SimpleQA Verified tests stored factual knowledge with 1,000 curated questions, corrected reference answers and revised grading instructions.

Definition: SimpleQA Verified is a benchmark of 1,000 short factual questions that tests parametric knowledge: facts recalled from a language model's stored parameters. It revises OpenAI's SimpleQA through question filtering, reference-answer correction and revised automated grading.

Key facts

The SimpleQA Verified paper, revised on 10 March 2026, describes a benchmark filtered from SimpleQA's 4,326 questions.
As of October 2026, Google's dataset card specifies evaluation without search or retrieval tools.
The public correction commit dated 9 March 2026 changes a Lilongwe question from latitude to longitude while retaining its numeric answer.

How is the question set revised?

The paper's curation method removes questions sharing a reference URL, then filters questions with similar meanings or overlapping wording. It also removes questions whose reference websites adopt the described content-use controls for Google, OpenAI or Anthropic. These controls use robots.txt, a file containing instructions for automated systems.

The authors then rebalance answer types and topics. Source reconciliation and manual checks address ambiguous questions and labeling errors. Because cleaning makes the benchmark easier, the final selection removes questions answered correctly by all the models used for difficulty filtering. The resulting set measures performance on deliberately challenging factual questions.

How are reference answers and grades corrected?

A reference answer is the expected answer used to grade a model's response. In the paper's source reconciliation process, the authors use search-assisted models and the original reference URLs to check conflicting evidence. They remove ambiguous non-numeric questions and correct answers or URLs where possible. Numeric questions undergo a separate consistency check using a 5% margin relative to the original reference answer. Manual review also corrects date precision and invalid or irrelevant URLs.

For grading, an autorater is a language model instructed to judge responses. The revised instructions replace a general significant-figure rule with question-specific acceptable numeric ranges. Small discrete counts can require an exact match. The autorater judges the part that directly answers the question and ignores additional information. A response listing several candidates must select one as more likely; otherwise, it receives a “not attempted” grade.

The grading scheme retains the labels correct, incorrect and not attempted. Its F1 score is the harmonic mean of accuracy across all questions and accuracy among attempted questions. This combines factual recall with the correctness of answers the model chooses to attempt.

What does a released correction look like?

Google's dataset change replaces “latitude” with “longitude” in the Lilongwe prompt. The following extract represents selected fields from the corrected dataset record:

{
  "original_index": 55,
  "problem": "What is the longitude of Lilongwe in decimal format?",
  "answer": "33.7738 (acceptable range: anything between 33.7586 and 33.8022)"
}

This is a correction to the question wording. The numeric answer and acceptable range remain unchanged. Recording the dataset revision therefore matters when comparing evaluations.

The dataset card explains that original_index links a record to the original SimpleQA question. Records also include reference URLs, a topic, an answer type and flags for reasoning and multi-step questions. These fields allow reviewers to trace an item and inspect its supporting material.

FAQ

Does SimpleQA Verified measure website citation performance?

The paper evaluates factual recall without tools. For a website team, its score describes performance under that knowledge-recall protocol; evaluating page selection and citation requires a separate assessment using the relevant search interface.

Can you use search during the evaluation?

The dataset card specifies a tool-free evaluation. The paper reports near-perfect performance with tools, which changes the task away from testing stored knowledge.

Can you compare its scores directly with SimpleQA scores?

The SimpleQA Verified paper changes both question selection and grading instructions. A comparison should identify the dataset revision and autorater configuration; its published comparison uses the same revised autorater for both question sets.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer