Mixture of Experts

A model design where only a few specialized sub-networks (experts) activate for each piece of input. Gives the capacity of a huge model at the running cost of a smaller one. Abbreviated MoE.

“It's a mixture of experts, so only a slice of the parameters fire per token.”