In short. Citation correctness evaluates support for a claim in its cited evidence, with separate checks for completeness and source quality.

Definition: Citation correctness measures whether a cited source supports its associated claim. For an AI answer, this means comparing the claim with the contents of the cited page or passage.

The starting point is Gao and colleagues' 2023 paper, Enabling Large Language Models to Generate Text with Citations. It introduces a benchmark for large language models (LLMs), called Automatic LLMs' Citation Evaluation (ALCE). ALCE evaluates answer correctness and citation quality separately.

Key facts

The ALCE paper evaluates whether cited passages support an answer and whether irrelevant passages appear among its citations.
Liu and colleagues' verifiability study defines citation precision as the proportion of citations that support their associated statements.
Rashkin and colleagues' Attributable to Identified Sources framework, abbreviated AIS, evaluates statements about the external world against provided evidence. It treats source quality as a complementary assessment.

How does ALCE test citation support?

In its citation evaluation, the ALCE paper divides generated prose into statements at sentence boundaries. Its evidence units are 100-word passages, and a statement can cite several passages.

Natural language inference is a test of whether evidence entails a statement, meaning that the statement follows from that evidence. ALCE uses the TRUE natural language inference model to check whether the combined cited passages support each statement. Citation recall averages these statement-level checks across the answer.

Its automatic citation precision method also examines individual citations. A citation is considered irrelevant when it cannot support the statement alone and removing it leaves the remaining citations able to support the statement. A citation receives precision credit only when the statement has full combined support. The method permits redundant citations that individually support the statement. The paper's appendix records a limitation: the automatic method can wrongly penalize a citation that provides partial support.

How does correctness differ from completeness?

Citation correctness concerns the support provided by an attached reference. Citation completeness concerns whether the answer's factual statements receive adequate cited support. The verifiability study measures these through citation precision and citation recall, respectively.

Its precision calculation counts citations that fully support their statement. It also counts partially supporting citations when the combined citations fully support the statement and none supplies full support alone. These rules matter when a sentence combines claims supported by different pages.

The AIS framework additionally separates attribution from evaluating the source itself. Its definition of attribution asks whether a reader can affirm a statement according to the supplied evidence. A review of source quality remains a separate part of assessing factual reliability.

How can a website citation be reviewed?

A manual review can follow the verifiability study's annotation method: identify the sentence associated with the citation, open the linked page, and compare its contents with the sentence's claims. The study uses these support categories:

Full support
Partial support
No support

Full support covers all information in the statement. Partial support covers some information while other parts are missing or contradictory. No support covers an irrelevant or contradictory page that supports none of the statement. When several citations accompany a sentence, the review also checks their combined support.

For a website owner, this produces a claim-level assessment of how the answer uses the page. A page can contribute evidence for part of a sentence while another source supplies the remaining evidence.

FAQ

Does a working link establish citation correctness?

You still need to compare the page with the associated claim. The verifiability study judges the contents of cited pages as providing full, partial, or no support.

Can several pages support a sentence together?

Yes. The ALCE method combines cited passages when checking statement support. Its citation precision calculation separately tests for irrelevant references.

Does supporting a claim establish source reliability?

The AIS framework evaluates attribution to supplied evidence. It places source quality among complementary assessments needed to evaluate factuality.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer