BrowseComp benchmark

Definition: BrowseComp is OpenAI's benchmark for evaluating browsing agents, AI systems that search and read the web to answer questions. It tests finding difficult factual answers by comparing short responses with reference answers.

Key facts

• The BrowseComp paper, submitted in April 2025, describes 1,266 questions collected by human trainers.
• As of 17 October 2026, OpenAI's public evaluation prompt requests an explanation, an exact answer and a confidence percentage.
• The ALCE paper evaluates citation quality separately from answer correctness, using measures of evidence support and citation relevance.

What makes the browsing tasks difficult?

The BrowseComp paper describes questions built backwards from a known fact. Trainers select a starting subject and combine identifying constraints so that finding the answer requires searching among many candidates. Once a candidate is found, checking it against the constraints is easier.

The intended challenge is sustained searching and changing search strategy when a route fails. The reference answer remains short, even when finding it involves substantial browsing.

How is an answer graded?

OpenAI's evaluation file supplies this response template:

Explanation: {your explanation for your final answer}
Exact Answer: {your succinct, final answer}
Confidence: {your confidence score between 0% and 100% for your answer}

These are output fields in an evaluation protocol. The explanation accompanies the answer, while the confidence field records the model's own estimate of its correctness. That estimate is separate from the grader's verdict.

The same file instructs a grading model to extract the final answer and compare it with the supplied reference. The rubric accepts matching answers, including numerical answers within a small margin of error. It directs the grader to focus on meaningful differences between answers and to avoid solving the question again. An explanation or a list of links therefore has no separate citation-quality criterion in this rubric.

What does a BrowseComp score leave out?

The BrowseComp paper identifies long responses and ambiguous user questions as tasks outside its chosen scope. Its experiments also combine 64 outputs per question using different answer-selection strategies. A comparison of scores needs the browsing effort and attempt-selection procedure alongside the model name.

Citation support means that cited evidence backs the statement attached to it. The ALCE paper measures this with citation recall, which checks support across answer statements. Its citation precision measure identifies irrelevant citations. ALCE averages recall across statements and precision across citations, making the evidence relationship an explicit evaluation target.

Applied to a website, this distinction matters: BrowseComp's answer-matching rubric provides no measure of a particular page's citation frequency or ranking. Evaluating a cited report requires inspecting the relationship between its claims and their evidence in addition to checking the final answer.

Related terms

• Agentic search
• ALCE benchmark
• Does a browsing benchmark score measure citation quality?
• GAIA benchmark

FAQ

Can I interpret a score as performance on ordinary customer questions?

BrowseComp's paper defines a deliberately difficult, short-answer task set. Testing ordinary customer questions requires an evaluation that includes their ambiguity and expected response format.

Does the confidence percentage validate the citations?

The public prompt asks for confidence in the final answer. Its grading rubric compares that answer with a reference; ALCE's citation measures instead inspect evidence support and relevance.

Sources

1. OpenAI, "BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents", 16 April 2025
2. OpenAI, "browsecomp_eval.py", repository snapshot retrieved 17 October 2026
3. Gao et al., "Enabling Large Language Models to Generate Text with Citations", EMNLP, December 2023

---

Check how a page reads to answer engines. Open the GEO score analyzer