Lost in the Middle paper

Definition: Lost in the Middle: How Language Models Use Long Contexts is a research paper that tests how the position of relevant information within an input context, the text supplied to a model, affects task accuracy. It reports that performance often falls when the information needed for an answer appears in the middle of the tested inputs.

Key facts

• The ACL Anthology record lists Nelson F. Liu and colleagues as authors and publication in Transactions of the Association for Computational Linguistics in 2024.
• The paper tests multi-document question answering and synthetic key-value retrieval, where a model returns the value associated with a supplied key.
• The authors' accompanying repository provides question-answering data with 10, 20 and 30 documents, plus instructions for generating examples with different answer-document positions.

How does the experiment isolate position?

In the question-answering experiment, the authors use 2,655 queries from NaturalQuestions-Open, a dataset pairing historical Google search queries with human-annotated answers from Wikipedia. Each input contains a question and a set of passages. Exactly one passage contains the annotated answer; the others serve as distractors.

The authors move the answer-bearing passage within the input while keeping the desired answer unchanged. Separately, they change context length by adding or removing distractor passages. Accuracy measures whether an annotated answer appears in the model's output. This design distinguishes the effect of document position from the effect of supplying more documents.

The key-value task removes much of the natural-language interpretation: the input contains a JSON object with randomly generated identifier strings as keys and values. The model receives a key and must return its associated value. The experiments use 75, 140 and 300 pairs.

What does the U-shaped curve show?

The results section describes a U-shaped accuracy curve in multi-document question answering. Accuracy is generally higher when the relevant passage appears at the beginning or end of the input, with lower accuracy for middle positions. The paper calls the preference for the beginning primacy bias and the preference for the end recency bias.

Task and model differences matter. Claude-1.3 and its extended-context counterpart perform nearly perfectly on the evaluated key-value retrieval settings. Other tested models struggle with middle positions. Placing the query both before and after the data produces near-perfect key-value retrieval across the tested models, while it changes question-answering trends only slightly.

The paper also compares models with their extended-context counterparts. When the same input fits both models' context windows, their question-answering performance is nearly identical.

How does this apply to a website or retrieval system?

The paper's evaluation recommendation is to measure how accuracy changes with the location of relevant information. For a system that retrieves website passages and supplies them to a model, this suggests testing passage order as part of the answering pipeline. The experiments provide no page-ranking rule or preferred location for facts within an HTML page.

The authors' README provides this command for generating inputs with 20 documents and varying the answer-bearing document's position:

for gold_index in 0 4 9 14 19; do
    python -u ./scripts/make_qa_data_from_retrieval_results.py \
        --input-path nq-open-contriever-msmarco-retrieved-documents.jsonl.gz \
        --num-total-documents 20 \
        --gold-index ${gold_index} \
        --output-path qa_data/nq-open-20_total_documents_gold_at_${gold_index}.jsonl.gz
done

These indices span the beginning, middle and end of the document sequence. The command uses the retrieval-results file specified in the README.

Related terms

• Context window
• Retrieval-augmented generation (RAG)
• Document chunking

FAQ

What does “middle” refer to?

It refers to the position of relevant information within the model's supplied input. In the paper's question-answering experiment, the authors change this position by reordering documents.

What should I measure when testing my own answering system?

You can follow the paper's protocol by moving the answer-bearing passage while holding the question and passage set fixed. Compare answer accuracy across positions, and evaluate changes in the number of supplied passages separately.

Sources

1. ACL Anthology, "Lost in the Middle: How Language Models Use Long Contexts", 2024
2. Transactions of the Association for Computational Linguistics, "Lost in the Middle: How Language Models Use Long Contexts", February 2024
3. Nelson F. Liu and contributors, "Lost in the Middle: How Language Models Use Long Contexts", README, undated

---

Check how a page reads to answer engines. Open the GEO score analyzer