In short. REALM jointly pretrains a retriever and a language encoder by rewarding retrieved passages that help predict masked words.

Definition: REALM: Retrieval-Augmented Language Model Pre-Training is a 2020 paper proposing a system that learns to retrieve external text while training a language model to predict masked words. The prediction task supplies the learning signal for the retriever, without human labels identifying useful documents.

Key facts

The PMLR publication record describes retrieval during pretraining, fine-tuning and inference.
Google Research's REALM explanation identifies Wikipedia as the external knowledge collection used in its experiments.
The released implementation's README explains that documents helping predict missing words receive more weight during training.

How does retrieval supply a pretraining signal?

Pretraining means learning language representations from unlabeled text before training for a specific task. Masked language modeling is a pretraining task in which the model predicts words hidden from an input passage.

The REALM paper couples a neural knowledge retriever with a knowledge-augmented encoder, a model that reads the input together with retrieved text. The retriever selects potentially helpful passages. The encoder uses those passages to predict the hidden words.

The retrieved document is a latent variable: training examples supply the missing words, without naming the document that should support their prediction. The training objective approximates a sum over the collection using the highest-scoring retrieved documents, combining their predictions with retrieval probabilities as weights. A document receives a positive retrieval update when it makes the correct prediction more likely than expected under the current retrieval distribution.

Google Research's worked example gives this masked input:

We paid twenty __ at the Buckingham Palace gift shop.

The target word is “pounds”. Supporting text can connect Buckingham Palace with London or identify the United Kingdom's currency. Several passages can therefore help with the same prediction.

The paper also masks named entities and dates to encourage knowledge-dependent predictions. It includes an empty candidate document for cases where external information is unnecessary. When the training passage also appears in the retrieval collection, that original document is excluded to prevent copying the unmasked sentence.

How does REALM search a large collection during training?

An embedding is a numerical vector representing text. The paper's retriever architecture encodes inputs and documents separately, then scores their relevance using the inner product of their vectors. Maximum Inner Product Search, or MIPS, finds candidates with high scores without running the encoder over the entire collection for every input.

Document vectors are precomputed in a search index. Training changes the document encoder, so the index needs refreshing. Google's technical explanation reports updating document vectors every 500 training steps. An asynchronous refresh lets training continue while a separate process rebuilds the index.

This index update concerns learned vector representations during pretraining. In the paper's question-answering fine-tuning experiments, document embeddings stay fixed while the input encoder continues to learn.

What does the evaluation measure?

Open-domain question answering requires answering a question without receiving a preselected supporting document. The paper's evaluation uses NaturalQuestions-Open, WebQuestions and CuratedTrec, measuring exact matches against reference answers.

After pretraining, REALM is fine-tuned with question-answer examples. Its answer model selects a contiguous span of text from a retrieved document. The training experiments compare Wikipedia and CC-News as pretraining text while using Wikipedia as the retrieval collection. These experiments evaluate question answering; the paper supplies no test of website changes or present-day answer-engine citation rates.

FAQ

Do you need labeled relevant documents to pretrain REALM?

The implementation README describes an unsupervised fill-in-the-blank objective. Missing-word prediction provides feedback about which retrieved documents help; subsequent task-specific fine-tuning uses supervised examples.

What does “document” mean in this paper?

The paper uses “document” for a passage from the knowledge collection. A retrieval candidate can be part of an article.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer