In short. The study measures whether AI search sentences have full source support and whether each citation supports its associated sentence.
Definition: Evaluating Verifiability in Generative Search Engines is a study by Nelson F. Liu, Tianyi Zhang and Percy Liang that audits citation support in AI search responses. It measures verifiability through citation recall, the proportion of checkable sentences fully supported by their citations, and citation precision, the proportion of citations that support their associated sentences.
Key facts
- The ACL Anthology record lists a December 2023 publication auditing Bing Chat, NeevaAI, perplexity.ai and YouChat.
- The paper's evaluation setup uses 1,450 queries per system and saves each system's first complete response, collected between late February and late March 2023.
- The authors' companion repository releases human evaluation annotations linking responses, sentences, citations and support judgments.
Which queries and responses does the audit cover?
The paper evaluates 12 query distributions. These include historical Google queries from NaturalQuestions, open-ended essay questions from the All Souls College entrance examination, model-generated debate questions, questions from Reddit's Explain Like I'm Five community, and keyword queries derived from WikiHow article titles.
For the live Reddit questions, the researchers query the systems immediately after a new post appears. This reduces the opportunity to retrieve the original discussion as an answer source. The audit records a single response per query, without follow-up conversation.
The systems also differ in how often they decline to answer. NeevaAI abstains on 22.7% of the evaluated queries, while each other system abstains on fewer than 0.5%. These response patterns matter when comparing results across systems.
How do annotators evaluate citation support?
In the evaluation method, a statement is a sentence, and its associated citations are the citation markers within that sentence. Annotators first identify sentences requiring verification: statements about the external world, including apparently obvious facts. Questions addressed to the user and statements about the system itself are examples excluded from this category.
Citation recall counts sentences fully supported by their associated sources. The support judgment follows the attributable to identified sources framework: a reader should be able to affirm the sentence according to the cited pages, within the context of the query and response.
Citation precision evaluates each citation as providing full, partial or no support. A partially supporting citation counts toward precision only when the sources together fully support the sentence and no individual citation fully supports it alone. In plain-text notation, the paper's calculations are:
citation recall = fully supported checkable sentences / all checkable sentences
citation precision = (fully supporting citations + qualifying partially supporting citations) / all citations
The second formula uses the joint-support rule described above. A sentence with several citations can therefore count as supported while some of its individual citations receive no precision credit.
The study involves 34 annotators. Each response receives one annotation; an agreement check collects three annotations for 250 randomly sampled query-response pairs. Annotators also rate fluency and perceived usefulness separately from citation support.
What does the study report, and how can its records be inspected?
The paper's results report average citation recall of 51.5% and citation precision of 74.5% across the audited systems. These figures describe the historical responses collected for this study.
The released annotation format records citation targets in link_target, sentence-level support in statement_supported, and individual citation judgments in citation_supports. Its evidence field holds annotator-selected supporting text for complete or partial support. For a website owner reviewing a linked page, these fields provide a concrete model for recording which answer sentence the page supports and the passage used to justify that judgment.
Related terms
FAQ
Does a citation beside a sentence count as full support?
Under the paper's method, you check whether the cited sources support the whole sentence. A page supporting only part of it receives a partial-support judgment.
Where can you examine the human judgments?
The authors' repository README links human_evaluation_annotations.jsonl and describes its fields. You can inspect the sentence support decision alongside the individual citation judgments and recorded evidence.
Sources
- Association for Computational Linguistics, "Evaluating Verifiability in Generative Search Engines", December 2023
- Nelson F. Liu, Tianyi Zhang and Percy Liang, "Evaluating Verifiability in Generative Search Engines" (paper), December 2023
- Nelson F. Liu and collaborators, "Evaluating Verifiability in Generative Search Engines" (repository README), undated
Check how a page reads to answer engines. Open the GEO score analyzer