← The glossary

inference

To begin with

When you ask an assistant a question, the model relearns nothing: it uses what it has already learned to compute an answer. That is inference.

The facts

During inference, the model computes an output from the inputs and its weights, without changing them. A language model generates its answer token by token and keeps a KV cache to avoid redoing some calculations. What you add to the conversation changes the context of the answer, not the weights.

Key takeaway

Inference uses the model; training changes it. A conversation does not retrain the model.

The stories about it

The notebook opens soon.The first stories are on their way.