Deep Learning

ReLU, explained

ReLU is an activation function that outputs zero for negative inputs and the input itself for positive inputs.
Back to Glossary

Part of the Semantic Notion technical glossary.

Short Definition

ReLU is an activation function that outputs zero for negative inputs and the input itself for positive inputs.

Intuition

ReLU keeps positive signals and shuts off negative ones, which makes it simple and efficient for deep networks.

Technical Definition

ReLU computes f(x) = max(0, x), introducing a piecewise linear nonlinearity with sparse activations.

Example

A hidden layer may apply ReLU after a matrix multiplication before passing values to the next layer.

Common Misunderstandings

ReLU can suffer from dead units when neurons stop activating.
ReLU is common but not the only useful activation.

Same Category

More from Deep Learning

Activation FunctionAn activation function is a nonlinear operation applied inside a neural network layer.Batch NormalizationBatch normalization normalizes layer activations using batch statistics to make training more stable.ConvolutionA convolution is an operation that applies a small learned filter across local regions of data.Deep LearningDeep learning is machine learning with neural networks that learn layered representations from data.EmbeddingAn embedding is a learned vector representation of an object such as a word, document, image, user, or product.Latent SpaceLatent space is the internal representation space where a model encodes hidden factors or learned structure.

Related Reading

Essays that use this concept

Machine Learning FoundationsBackpropagation Explained Without Hand-Waving

Start here

Need the broader concept map?

Return to the glossary index to move from this definition into adjacent AI, ML, mathematics, and computer science terms.

Browse Glossary