In short. GopherCite combines quoted evidence with configurable abstention, while its evaluation distinguishes source support from factual truth.
Definition: Teaching language models to support answers with verified quotes is the research paper introducing GopherCite, a question-answering model that supplies verbatim source quotations with its answers. The experiment also uses a score threshold to decline questions whose candidate answers receive low scores.
Key facts
- The paper's abstract reports high-quality answers on filtered Natural Questions and ELI5 question sets at rates of 80% and 67%, respectively.
- The full paper describes constrained generation that keeps quotations verbatim from the claimed source.
- DeepMind's research announcement shows that evidence-supported answers can still repeat misconceptions on adversarial questions.
How does GopherCite produce evidence-backed answers?
The paper's methods describe self-supported question answering: generating an answer together with evidence supporting its claims. The system forwards the input question to Google Search, supplies retrieved document text to the model, generates candidate responses, and selects a response using a reward model. A reward model is a model trained to predict human preferences between candidate responses.
Human participants compare answers partly on how well the evidence supports them. Their ratings inform supervised training and reinforcement learning from human preferences, a training method that optimizes the reward model's score.
The paper gives this inline evidence template:
%<Claim>%(Document title)%[Quote from document]%
Constrained generation makes the quote match text in the retrieved document. The format also allows software to separate the claim and evidence for display. Readers can inspect the specific passage used to support an answer.
What does abstention change in the experiment?
Abstention means declining to answer a question. The paper's abstention method applies a global threshold to the reward model's score and returns “I don’t know” when the score falls below it. The underlying model is trained to answer directly; the threshold supplies the configurable refusal mechanism.
Coverage is the proportion of questions attempted. At 70% coverage, the experiment reports more than 90% supported-and-plausible answers on filtered Natural Questions and more than 80% on filtered ELI5. These percentages describe the attempted questions. Changing the threshold changes both coverage and the quality measured among the answers returned.
What do the reported scores measure?
The evaluation protocol defines a plausible answer as a reasonable, on-topic response. A supported answer has accompanying evidence sufficient to verify its validity under the rating instructions. The reported comparison uses overlapping subsets of 115 Natural Questions questions and 121 ELI5 questions with ratings across the compared systems.
The paper separately evaluates truthfulness on TruthfulQA, a benchmark containing questions that elicit misconceptions. It finds that quotations about fiction, metaphor or satire can support literally false claims. For website readers, this separates checking whether a quote matches a page from checking whether the quoted passage establishes the answer's factual accuracy.
Related terms
FAQ
Does “verified quotes” mean every answer is true?
The paper's constrained sampling verifies that quotations come verbatim from the claimed source. Its TruthfulQA evaluation shows that a supported answer can still be literally false.
What should I inspect when an answer includes a quotation?
You can compare the answer with the quoted passage and examine its context. DeepMind's announcement explains that some claims require multiple pieces of evidence and a logical argument connecting that evidence to the claim.
Sources
- arXiv, "Teaching language models to support answers with verified quotes", 21 March 2022
- DeepMind, "Teaching language models to support answers with verified quotes", 16 March 2022
- Google DeepMind, "GopherCite: Teaching language models to support answers with verified quotes", 16 March 2022
Check how a page reads to answer engines. Open the GEO score analyzer