← The glossary

router

To begin with

In a MoE model, each token does not go through all the experts. Like a railway signalman, the router decides which track it takes.

The facts

The router is a small layer that computes a score for each expert from the token’s representation, then keeps the best ones: two out of eight in Mixtral. Its coefficients are learned during training and stored in the weights. Its choices, however, are recomputed for every token and are not saved in the model.

Key takeaway

The router learns a rule for choosing, not a list of answers. The rule is saved; the choices are made again every time.

The stories about it

The notebook opens soon.The first stories are on their way.