BEIR benchmark

Definition: BEIR (Benchmarking IR) is a benchmark for evaluating information retrieval, the task of finding relevant documents for a query. It tests how retrieval methods transfer across different tasks and subject domains without training on the target task or domain.

Key facts

• The BEIR paper, published at NeurIPS in 2021, introduces 18 English evaluation datasets spanning nine retrieval tasks.
• The Resources for Brewing BEIR preprint describes aggregate scores as macro-averages: each dataset score receives equal weight.
• As of 17 October 2026, the official repository provides a Python evaluation framework and a SciFact example that retrieves documents and evaluates the results against relevance judgements.

What makes the evaluation heterogeneous?

Heterogeneous evaluation varies the retrieval problem as well as the subject matter. The Brewing BEIR preprint describes Natural Questions as finding Wikipedia passages that answer questions, while ArguAna concerns argument retrieval and SciFact concerns fact checking. Queries range from a few keywords to paragraph length.

These differences matter when selecting an evaluation set for a website's search or document retrieval system. A question about a scientific claim and a keyword query over news articles represent different retrieval requirements. The benchmark makes those differences visible through separate dataset results.

How is a BEIR run scored?

The original paper uses nDCG@10, a ranking metric that accommodates binary relevance labels and graded relevance judgements, evaluated at a cutoff of ten results. Binary labels distinguish relevant from irrelevant documents; graded labels express levels of relevance.

Reference relevance judgements, called qrels, provide the evaluation labels. The repository's SciFact example loads a document collection, queries and judgements for the test split. These lines are an excerpt from that example; retriever is the configured retrieval model wrapper:

corpus, queries, qrels = GenericDataLoader(data_folder=data_path).load(split="test")
results = retriever.retrieve(corpus, queries)
ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values)

The same example saves the retrieved ranking and evaluation results to files. Keeping those outputs makes it possible to inspect which documents produce a score and to reuse the ranking for reranking, which reorders retrieved documents.

What are the transfer limits?

The BEIR paper finds that performance on the training domain correlates poorly with performance across other domains. Its experiments retain BM25, a keyword-based retrieval method, as a strong baseline. These are findings about the evaluated models in the paper's experimental setting.

Relevance judgements also affect comparisons. In the paper's TREC-COVID analysis, adding missing judgements improves scores for methods that retrieve relevant documents outside the original annotation pool. A benchmark score therefore depends on the reference annotations as well as the retrieval method.

The Brewing BEIR preprint identifies another comparison problem: papers report results on partially overlapping dataset subsets. Its dataset-level visualizations help locate gains and losses relative to a baseline. A practical implication is to compare the same datasets and examine results closest to the intended use case before choosing a retrieval method for a website. A separate evaluation of an answer engine's outputs is needed to assess source selection and citation quality.

Related terms

• Sparse retrieval
• Dense retrieval
• Retrieval reranking
• Dense Passage Retrieval paper

FAQ

Does zero-shot mean the model has never been trained?

The Brewing BEIR preprint defines the setting by the absence of training data or supervision for the target tasks and domains. You can evaluate a previously trained model for transfer under those conditions.

Can I use BEIR with my own documents?

The official repository supports preprocessing a custom retrieval dataset. You need a document collection, queries and relevance judgements to evaluate whether retrieved documents match those judgements.

Sources

1. NeurIPS, "BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models", 2021
2. arXiv, "Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard", 13 June 2023, version 1
3. BEIR maintainers, "BEIR repository README", accessed 17 October 2026

---

Check how a page reads to answer engines. Open the GEO score analyzer