# REALM: Retrieval-Augmented Language Model Pre-Training

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang

2020 | arXiv:2002.08909

Source: https://arxiv.org/abs/2002.08909

Reading scope: abstract-and-metadata

Editorial update: 2026-10-20

Retrieval becomes part of language-model pretraining rather than only a downstream add-on.

## Why this matters

It makes learning access to external documents a central design question.

## The proposal

REALM addresses knowledge held implicitly in model parameters. The authors introduce a document retriever used during pretraining, fine-tuning and inference.

## What was inspected

The abstract describes unsupervised retriever learning through masked language modeling and reports open-domain question-answering improvements. This reading does not validate those comparisons independently.

## Editorial interpretation

The useful distinction is where retrieval is learned. REALM invites a comparison between knowledge access trained with the model and retrieval attached after training.

## Mechanism

Retrieve latent documents.

Attend to retrieved knowledge.

Learn retrieval using masked-language-model supervision.

## Limits

Training and indexing details need full-text inspection.

Reported QA results do not establish universal corpus transfer.