Short Definition
Active parameters are the subset of a model's parameters used for a specific input, token, or inference step.
Intuition
A sparse model can be enormous in total size while using only part of itself for each token.
Technical Definition
Active parameter count measures the parameters participating in the forward computation for a given route, distinct from total stored parameters.
Example
A MoE model may have hundreds of billions of total parameters but tens of billions active per token.
Common Misunderstandings
Active parameter count is not the same as memory requirement.
A lower active count does not automatically mean lower end-to-end latency.