pretraining
To begin with
Before it can follow instructions, a language model must first learn the language itself. This is the first and longest phase: pretraining.
The facts
During pretraining, the model reads huge amounts of text and often learns to predict the next token. The right answers come from the text itself: nobody has to write them by hand. This phase builds the basic capabilities and costs a great deal of computation. The resulting model then serves as the starting point for fine-tuning.
Key takeaway
Pretraining teaches a model to continue a text, not yet to follow instructions. That comes later, with adaptation.
The stories about it
The notebook opens soon.The first stories are on their way.