Short Definition
Mixture of Experts is a model architecture that routes inputs to a subset of specialized expert networks instead of activating every parameter.
Intuition
The model carries many specialists, but only a few are consulted for each token or example.
Technical Definition
MoE layers use a router or gating function to select expert subnetworks conditionally, increasing total capacity while controlling active compute.
Example
Inkling has many total parameters but activates only a subset for each token through expert routing.
Common Misunderstandings
Total parameters and active parameters are different in sparse MoE models.
MoE adds routing and load-balancing complexity.