router
To begin with
In a MoE model, each token does not go through all the experts. Like a railway signalman, the router decides which track it takes.
The facts
The router is a small layer that computes a score for each expert from the token’s representation, then keeps the best ones: two out of eight in Mixtral. Its coefficients are learned during training and stored in the weights. Its choices, however, are recomputed for every token and are not saved in the model.
Key takeaway
The router learns a rule for choosing, not a list of answers. The rule is saved; the choices are made again every time.
The stories about it
The notebook opens soon.The first stories are on their way.