Why this idea matters
It makes learning access to external documents a central design question.
The proposal
REALM addresses knowledge held implicitly in model parameters. The authors introduce a document retriever used during pretraining, fine-tuning and inference.
What was inspected
The abstract describes unsupervised retriever learning through masked language modeling and reports open-domain question-answering improvements. This reading does not validate those comparisons independently.
Editorial interpretation
The useful distinction is where retrieval is learned. REALM invites a comparison between knowledge access trained with the model and retrieval attached after training.
The mechanism, in brief
- Retrieve latent documents.
- Attend to retrieved knowledge.
- Learn retrieval using masked-language-model supervision.
Limits of this note
- Training and indexing details need full-text inspection.
- Reported QA results do not establish universal corpus transfer.
Continue reading
Related research paths ↗To examine another paper and keep working notes, open the Paper desk. Those notes stay in your browser until you export them.