Deep Learning

Activation Function, explained

An activation function is a nonlinear operation applied inside a neural network layer.
Back to Glossary

Part of the Semantic Notion technical glossary.

Short Definition

An activation function is a nonlinear operation applied inside a neural network layer.

Intuition

Without nonlinear activations, many stacked layers would collapse into something no more expressive than a single linear transformation.

Technical Definition

Activation functions transform intermediate layer outputs elementwise or channelwise, enabling neural networks to approximate nonlinear functions.

Example

ReLU sets negative values to zero and leaves positive values unchanged.

Common Misunderstandings

Activation functions are not just cosmetic; they define what functions the network can learn.
Different activations affect optimization and gradient flow.

Same Category

More from Deep Learning

Batch NormalizationBatch normalization normalizes layer activations using batch statistics to make training more stable.ConvolutionA convolution is an operation that applies a small learned filter across local regions of data.Deep LearningDeep learning is machine learning with neural networks that learn layered representations from data.EmbeddingAn embedding is a learned vector representation of an object such as a word, document, image, user, or product.Latent SpaceLatent space is the internal representation space where a model encodes hidden factors or learned structure.Neural NetworkA neural network is a machine learning model built from layers of connected computations that transform inputs into outputs.

Related Reading

Essays that use this concept

Machine Learning FoundationsBackpropagation Explained Without Hand-Waving

Start here

Need the broader concept map?

Return to the glossary index to move from this definition into adjacent AI, ML, mathematics, and computer science terms.

Browse Glossary