In short. Use reviewed, matched question sets and record query, instruction, answer and source languages separately when comparing AI answers.
Definition: Comparing answers across languages means evaluating equivalent questions under recorded language conditions. A matched question set preserves the information requested across translations, while the evaluation records the language requested for the answer and the languages of its sources separately.
Key facts
- The MIRACL paper evaluates retrieval across 18 languages, with questions and searched documents in the same language and relevance judgments from native speakers.
- MKQA, a multilingual question answering benchmark, aligns 10,000 question-answer pairs across 26 languages, enabling comparison on equivalent questions.
- Chirkova and colleagues' multilingual retrieval study measures correct output language separately from answer quality and examines alternative spellings of names.
- As of 17 October 2026, Google's language guidance distinguishes the interface language from result-language selection and identifies location as a factor in result language.
How should the question sets match?
A matched set assigns a shared identifier to each information need and retains it across language versions. MKQA's paper describes parallel questions as the same examples in every language and uses expert bilingual translators with localization criteria. For an agency, this suggests a reviewed translation set with the same intent, entities, constraints and expected factual answer.
A bilingual reviewer should check each version before collection. A translation that changes the place, product or eligibility condition belongs in a separate comparison. Market-specific questions can form an additional panel, clearly labeled by market; their results should be reported separately from the matched language comparison. This preserves the distinction between translating a question and changing the information need.
Which language conditions belong in the test record?
Chirkova and colleagues distinguish query language from the language of the searched collection and test explicit answer-language instructions. Their prompt template includes this instruction fragment, where the placeholder represents the target language:
Answer in {UL}.
An agency can record the question language, instruction language, requested answer language and observed source languages in separate fields. A source-language restriction, where available, also belongs in the record. In a controlled document collection, the collection's language is an explicit experimental condition.
Google's language guidance explains that interface preferences influence result language alongside location; its language filters mainly apply to web results. This supports recording interface language, region and available filters separately. The collection log should also retain the engine and displayed model, collection time, session conditions, exact prompt, full answer and linked source URLs. Keeping these conditions consistent across language versions makes the comparison easier to audit.
How should the agency evaluate the answers?
The multilingual retrieval study defines correct language rate as the share of outputs written in the user's language. It also addresses spelling variation in named entities, meaning names of people, places or organizations. These provide separate checks for language compliance and factual content.
For an agency report, each matched question can carry a factual assessment, an observed answer language and a list of cited pages. A reviewer should check whether linked pages support the answer's claims and accept established local spellings of the same entity. Results should show the assessed question count for each language and identify missing answers. A comparison table can then separate factual errors, language mismatches and differences in linked sources without treating every textual difference as an error.
Related terms
FAQ
Can I use MIRACL as a ready-made answer comparison?
MIRACL evaluates retrieving relevant passages from a same-language collection, as its paper explains. For an agency answer audit, use its explicit retrieval conditions as a methodological reference and define a separate answer assessment.
Can I translate a question list automatically?
You can use automatic translation as a draft, followed by bilingual review. MKQA uses expert bilingual translation with localization criteria, providing a reference for preserving equivalence across language versions.
Sources
- MIT Press / ACL Anthology, "MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages", 2023
- MIT Press / ACL Anthology, "MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering", 2021
- Association for Computational Linguistics, "Retrieval-augmented generation in multilingual settings", 16 August 2024
- Google Search Help, "How Google knows what language to show in search results", undated, accessed 17 October 2026
Check how a page reads to answer engines. Open the GEO score analyzer