In short. ALCE evaluates fluency, answer correctness and citation quality for systems that retrieve evidence and generate cited answers.
Definition: ALCE, short for Automatic LLMs' Citation Evaluation, is a benchmark for evaluating systems that retrieve evidence and generate answers with citations. It measures fluency, answer correctness and citation quality separately.
The paper Enabling Large Language Models to Generate Text with Citations, by Tianyu Gao, Howard Yen, Jiatong Yu and Danqi Chen, introduces ALCE at EMNLP in December 2023.
Key facts
- The paper's abstract describes an end-to-end task: systems retrieve supporting evidence and generate answers with citations.
- The published paper evaluates citation recall, meaning whether statements have full support, and citation precision, which detects irrelevant citations.
- The authors' ALCE repository provides code and data for ASQA, QAMPARI and ELI5, alongside implementations of the paper's baselines.
What does end-to-end evaluation involve?
In Section 2 of the paper, a system receives a question and a corpus, meaning a collection of text passages. It retrieves relevant passages, writes a response and attaches citations to supporting passages. The benchmark divides documents into 100-word passages.
The evaluation treats sentences as statements, except in QAMPARI, where each listed entity counts as a statement. A statement can cite several passages. This setup tests evidence retrieval and answer construction together.
Which datasets does ALCE use?
The dataset description specifies 1,000 examples sampled from each dataset's development set.
| Dataset | Answer task | Retrieval corpus |
|---|---|---|
| ASQA | Resolve ambiguous factual questions in a long answer covering different interpretations. | A Wikipedia snapshot dated 20 December 2018. |
| QAMPARI | Return a list of entities whose answers come from different passages. | The same Wikipedia snapshot. |
| ELI5 | Answer explanatory questions from Reddit's Explain Like I'm Five forum. | Sphere, a filtered collection from Common Crawl. |
For ELI5, the paper also specifies that Reddit pages are excluded from the retrieval corpus. ALCE supplies evaluation examples without training data containing supervised citation annotations.
How are answers and citations scored?
Section 3 uses MAUVE, a text evaluation metric, to check fluency for ASQA and ELI5. QAMPARI's list answers are excluded from that fluency check.
Correctness depends on the dataset. ASQA measures how many reference short answers appear in the response. QAMPARI compares predicted entities with reference answers and gives full recall credit when at least five correct answers appear. ELI5 measures coverage of claims generated from reference answers.
For citation quality, ALCE uses TRUE, a natural language inference model. Natural language inference tests whether evidence supports a statement. Citation recall averages whether each statement has citations whose combined passages fully support it. Citation precision checks individual references: a citation is irrelevant when it fails to support the statement alone and removing it leaves full support from the remaining citations. A citation receives precision credit only when its statement also passes the recall check.
The paper's limitations section identifies the inference model's accuracy as a constraint on these automatic citation scores.
What does an evaluation run look like?
As of 17 October 2026, the repository instructions give this command for evaluating an ELI5 result file:
python eval.py --f {path/to/result/file} --citations --claims_nli --mauve
The flags request citation evaluation, claim coverage and fluency measurement. The evaluator saves results with a .score suffix. The repository also distributes precomputed retrieval results, allowing evaluation with supplied evidence passages.
Related terms
FAQ
Does ALCE check whether an answer addresses the question?
Yes. The correctness evaluation compares responses with reference answers or reference-derived claims. Citation evaluation separately checks support from the passages attached to the response.
Can I use ALCE with existing retrieval results?
Yes. The repository's data instructions provide the top 100 retrieved passages per question for ASQA, QAMPARI and ELI5, alongside code for reproducing retrieval.
Sources
- ACL Anthology, "Enabling Large Language Models to Generate Text with Citations", December 2023
- Association for Computational Linguistics, "Enabling Large Language Models to Generate Text with Citations", full paper, December 2023
- Princeton NLP, "ALCE repository README", accessed 17 October 2026
Check how a page reads to answer engines. Open the GEO score analyzer