In short. KILT assesses knowledge-intensive language tasks using a fixed Wikipedia source and scores that combine answer performance with evidence retrieval.

Definition: The KILT benchmark, introduced in KILT: a Benchmark for Knowledge Intensive Language Tasks (2021), evaluates language tasks that require information from a large knowledge source. It assesses task outputs alongside provenance, the Wikipedia evidence supporting those outputs.

Key facts

The published paper brings together eleven datasets spanning five tasks, all grounded in a shared Wikipedia snapshot.
As of 17 October 2026, the project README marks wikipedia_id as mandatory within provenance records.
The released evaluator awards KILT answer credit only when a complete reference evidence set ranks at the top.

What makes a task knowledge-intensive?

A knowledge-intensive task requires access to a large body of information to produce a specific output. The KILT paper includes fact checking, which evaluates a claim against evidence; entity linking, which assigns a Wikipedia page to an entity mentioned in text; and slot filling, which supplies an entity's missing relation value. It also includes open-domain question answering, where a system searches a knowledge source for an answer, and dialogue grounded in that source.

The inputs differ accordingly: a claim, a text passage, a structured query, a question or a conversation. FEVER supplies fact-checking examples, while Wizard of Wikipedia supplies dialogue examples. A common source makes it possible to compare systems across these different uses of knowledge without rebuilding an index for each dataset.

That source is the Wikipedia snapshot of 1 August 2019, containing 5.9 million articles. The authors map the datasets' original evidence into this fixed collection.

What does provenance look like?

The project README's data format associates an input with an output list. An output can contain an answer, provenance or both. Provenance records identify Wikipedia pages and can locate evidence through paragraph identifiers and character offsets.

This schematic output object follows those documented fields; the ellipses stand for values:

{
    'answer': ...,
    'provenance': [
        {
            'wikipedia_id': ...,
            'start_paragraph_id': ...,
            'start_character': ...,
            'end_paragraph_id': ...,
            'end_character': ...
        }
    ]
}

The paper's provenance definition allows evidence spans ranging from an entity to a whole article. Several spans can form a complementary evidence set: together, they supply the information needed for the output. The annotations can also record alternative valid evidence sets.

How does KILT evaluate answers and evidence?

The evaluation section of the paper distinguishes answer performance from evidence retrieval. Retrieval means finding relevant material in the knowledge source. The published retrieval evaluation operates at Wikipedia page level, even though provenance annotations can identify smaller spans.

For evidence, R-precision is the fraction of relevant pages among the first R retrieved pages, where R is the number of pages in a reference evidence set. When alternative sets exist, KILT takes the highest R-precision across them. HotpotQA requires two evidence pages, so retrieving a single relevant page leaves its evidence set incomplete.

For answers, Exact Match measures agreement with a reference answer after normalization. KILT-EM awards that answer credit only when R-precision equals 1: a complete reference evidence set must occupy the top positions. A correct answer paired with an incomplete evidence set therefore receives zero KILT-EM credit.

The released evaluator implements this gate using wikipedia_id. Its answer normalization lowercases text and removes punctuation, articles and extra whitespace. Reading an Exact Match result alongside KILT-EM distinguishes answer matching from answer matching accompanied by the required evidence pages.

FAQ

Can a correct answer receive zero KILT-EM credit?

Yes. The KILT scoring rule awards answer credit only when a complete reference evidence set ranks at the top. Answer correctness alone is insufficient.

Does KILT include questions with no available evidence?

The same paper defines KILT as an in-knowledge-base resource: evidence needed for every instance is present somewhere in its knowledge source. Unanswerable instances are outside that benchmark setting.

Can I use KILT to measure my website's citations in AI search?

KILT evaluates outputs and evidence against its fixed Wikipedia collection. Its scope supplies no measurement of how often a website appears in an answer engine's live results.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer