Mathematics for ML

Entropy, explained

Entropy measures uncertainty or information content in a probability distribution.
Back to Glossary

Part of the Semantic Notion technical glossary.

Short Definition

Entropy measures uncertainty or information content in a probability distribution.

Intuition

A predictable distribution has low entropy. A spread-out uncertain distribution has high entropy.

Technical Definition

Shannon entropy is the expected negative log probability of outcomes from a distribution.

Example

A fair coin has higher entropy than a coin that almost always lands heads.

Common Misunderstandings

Entropy is not disorder in a vague sense; in ML it is a precise quantity over probabilities.
Higher entropy is not always better or worse.

Same Category

More from Mathematics for ML

Cosine SimilarityCosine similarity measures how similar two vectors are by comparing their direction rather than their magnitude.Cross-EntropyCross-entropy measures how well one probability distribution matches another, often a model prediction against a target label.Dot ProductThe dot product combines two vectors into a single number that reflects their alignment and magnitude.EigenvalueAn eigenvalue is the scaling factor applied to an eigenvector by a linear transformation.EigenvectorAn eigenvector is a nonzero vector whose direction stays the same under a linear transformation.Euclidean DistanceEuclidean distance is the straight-line distance between two points in a vector space.

Related Reading

Essays that use this concept

Mathematics for MLWhat Is Entropy in Information Theory and Machine Learning?

Start here

Need the broader concept map?

Return to the glossary index to move from this definition into adjacent AI, ML, mathematics, and computer science terms.

Browse Glossary