← The glossary

expert

To begin with

Instead of running the whole model for every token, a MoE model shares the work among several computation blocks. Each of these blocks is an expert.

The facts

In a MoE layer, several experts sit side by side. For each token, the router picks a few of them: two out of eight in Mixtral. All experts are stored in the weights and must be available in memory, but only the chosen ones compute. Their specialisation is learned; it does not necessarily match a topic we could name.

Key takeaway

An expert is not a small model specialised in one topic. It is a computation block chosen token by token.

The stories about it

The notebook opens soon.The first stories are on their way.