Generalization through Memorization: Nearest Neighbor Language Models

Why this idea matters

A precursor showing that an external datastore can participate in prediction.

The proposal

The authors combine a pretrained language model with nearest-neighbor lookup. The external text collection supplies an additional prediction signal without retraining the neural model.

What was inspected

The abstract reports language-modeling and domain-adaptation results. We have not checked the experimental tables, so this note makes no independently verified performance claim.

Editorial interpretation

Read this as a memory-design branch of the atlas. It helps separate storing examples outside model weights from retrieving readable passages for an answer.

The mechanism, in brief

1. Represent context in the pretrained model space.
2. Find neighboring examples.
3. Interpolate neighbor and neural predictions.

Limits of this note

• This note does not audit datastore leakage or retrieval cost.
• Token prediction is different from document-grounded answering.

Continue reading

To examine another paper and keep working notes, open the Paper desk. Those notes stay in your browser until you export them.