In short. GAIA evaluates assistant task completion through short factual answers, including tasks that require web evidence and tools.
Definition: GAIA: a benchmark for General AI Assistants introduces GAIA, an evaluation of assistants answering factual questions through reasoning, tool use and the handling of different kinds of media. Tasks can require web browsing or reading attached files, and have short, verifiable answers.
Key facts
- The authors' original abstract describes 466 questions, with answers to 300 withheld for leaderboard evaluation.
- The ICLR paper defines three difficulty levels using the steps and tools needed by human annotators as a proxy.
- As of 17 October 2026, the official dataset card describes gated access and asks users to avoid sharing validation or test questions in a crawlable format.
What does a GAIA task require?
The paper's task description explains that questions involve finding and transforming information from supplied documents or the open web. Handling multiple media means working with material such as images, audio and spreadsheets alongside text.
Figure 1 gives a web research example: finding the actual enrolment count of a clinical trial on the NIH website. Figure 2 gives a file task: calculating a fast-food chain's food sales from an attached spreadsheet while excluding drinks. These examples make the role of evidence concrete: an assistant has to locate the relevant information and perform the requested operation.
The paper leaves tool selection open and allows different routes to an answer. Its web tasks require browsing through clicks; actions such as posting comments or booking meetings fall outside its tested capabilities.
How do the difficulty levels work?
The difficulty definitions in the paper describe Level 1 as generally needing no tool, or at most one tool with no more than 5 steps. Level 2 generally involves roughly 5 to 10 steps and a combination of tools. Level 3 allows arbitrarily long action sequences and any number of tools.
These are approximate categories. The authors explain that complex web navigation can put a question in Level 3 even when an annotator needs fewer than 10 steps. The labels therefore summarize the annotated task difficulty rather than imposing an execution limit on an assistant.
What does the score measure?
The evaluation section describes comparison with a reference answer, allowing normalization according to answer type. The paper's prompt uses this final-answer template:
FINAL ANSWER: [YOUR FINAL ANSWER]
Answers can be a number, a short string or a comma-separated list. As of 17 October 2026, the accompanying leaderboard scorer removes currency signs, percent signs and commas when normalizing numeric answers. Its default string comparison removes whitespace and punctuation and converts text to lowercase.
The scorer receives the model answer and the reference answer. It evaluates their agreement without inspecting cited pages. Consequently, a GAIA score reports final-answer correctness; assessing whether a linked page supports a claim requires a separate citation assessment.
Related terms
FAQ
Does every question require web browsing?
The paper includes tasks involving supplied files and allows Level 1 questions without tools. It also notes that assistants can use different combinations of capabilities to reach an answer.
What should you supply when evaluating an assistant?
The task instructions in the paper call for the question and any attached evidence, using zero-shot prompting, meaning without worked examples in the prompt. The dataset card identifies attachments through file fields alongside the question.
Sources
- Mialon et al., arXiv, "GAIA: a benchmark for General AI Assistants", 21 November 2023
- Mialon et al., ICLR, "GAIA: a benchmark for General AI Assistants", 2024
- GAIA authors, Hugging Face, "GAIA dataset", accessed 17 October 2026
- GAIA authors, Hugging Face, "scorer.py", accessed 17 October 2026
Check how a page reads to answer engines. Open the GEO score analyzer