Prompt test set

Definition: A prompt test set is a fixed collection of questions or instructions reused to evaluate an AI assistant under specified conditions. Each item has a grading rule, such as comparison with a reference answer, so results can be compared across runs.

Key facts

• OpenAI's BrowseComp paper describes a dataset of 1,266 questions with short, evidence-supported answers designed to remain stable over time.
• Stanford CRFM's HELM evaluation methodology controls how models receive tasks and evaluates them on shared scenarios to support comparison.
• OpenAI's BrowseComp evaluator, in its May 2025 revision, supports repeating the question collection through the n_repeats parameter.

How does BrowseComp construct its questions?

The BrowseComp methodology starts with human trainers writing fact-seeking questions. Trainers begin with a known fact, select characteristics of a person, event or artifact, and turn those characteristics into an inverted question: finding the answer requires searching, while checking a proposed answer is comparatively simple.

The selection process includes checks against existing models and five simple Google searches.

For a website evaluation, this approach suggests keeping supporting evidence alongside each reference answer and reviewing ambiguous questions before fixing the collection for repeated use.

What stays fixed between evaluation runs?

Stanford CRFM's HELM methodology distinguishes the task scenario from the procedure that turns it into a model prompt. Its scenario description includes the task, domain and language. HELM uses simple, generic prompts and reports that prompting strategy can substantially affect results.

Applied to a website's question set, this supports preserving exact question wording, language, prompt instructions and grading rules. A revised question belongs in a separately identified version of the set. Run records should identify the assistant or model tested and the date, so a later comparison has an explicit context.

HELM also exposes raw prompts and model predictions beneath aggregate results. Following that approach, keeping individual responses makes it possible to inspect which questions account for a changed result. Selecting relevant customer questions is part of defining the evaluation scenario.

What does a test prompt look like?

OpenAI's BrowseComp evaluator separates the question from a shared response template. The following reproduces that template's structure, with placeholders rather than a benchmark question:

{Question}

Your response should be in the following format:
Explanation: {your explanation for your final answer}
Exact Answer: {your succinct, final answer}
Confidence: {your confidence score between 0% and 100% for your answer}

The implementation keeps the reference answer separate from the question sent to the assistant. Its grader receives the question, response and reference answer. This separation provides a concrete pattern for storing a test item and evaluating its output without supplying the expected answer in the test prompt.

Related terms

• AI answer reproducibility
• Query sampling bias
• Citation tracking
• Citation correctness

FAQ

Does a difficult set represent customer searches?

The BrowseComp paper describes a limited subset of browsing capability and leave its relationship to open-ended user queries uncertain. You should select questions for the audience and tasks your evaluation covers.

Can I repeat the same questions?

The BrowseComp evaluator uses n_repeats to repeat its collection. You can use repeated runs to retain multiple responses to each fixed question.

How are short answers checked?

BrowseComp uses an AI grader to compare a predicted answer with a reference answer for semantic equivalence. You need a specified grading rule alongside the question collection.

Sources

1. OpenAI, "BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents", 16 April 2025
2. Stanford CRFM, "Language Models are Changing AI: The Need for Holistic Evaluation", 17 November 2022
3. OpenAI, "browsecomp_eval.py", 12 May 2025 revision

---

Check how a page reads to answer engines. Open the GEO score analyzer