expert
To begin with
Instead of running the whole model for every token, a MoE model shares the work among several computation blocks. Each of these blocks is an expert.
The facts
In a MoE layer, several experts sit side by side. For each token, the router picks a few of them: two out of eight in Mixtral. All experts are stored in the weights and must be available in memory, but only the chosen ones compute. Their specialisation is learned; it does not necessarily match a topic we could name.
Key takeaway
An expert is not a small model specialised in one topic. It is a computation block chosen token by token.
The stories about it
The notebook opens soon.The first stories are on their way.