← The glossary

pretraining

To begin with

Before it can follow instructions, a language model must first learn the language itself. This is the first and longest phase: pretraining.

The facts

During pretraining, the model reads huge amounts of text and often learns to predict the next token. The right answers come from the text itself: nobody has to write them by hand. This phase builds the basic capabilities and costs a great deal of computation. The resulting model then serves as the starting point for fine-tuning.

Key takeaway

Pretraining teaches a model to continue a text, not yet to follow instructions. That comes later, with adaptation.

The stories about it

The notebook opens soon.The first stories are on their way.