Transformers and LLMs

Attention, explained

Attention is a neural network mechanism that lets a model decide which parts of an input are most relevant when producing a representation or output.
Back to Glossary

Part of the Semantic Notion technical glossary.

Short Definition

Attention is a neural network mechanism that lets a model decide which parts of an input are most relevant when producing a representation or output.

Intuition

Instead of compressing everything into one fixed summary, attention lets the model look back at useful pieces of context when it needs them.

Technical Definition

Attention computes weights over values, usually from query-key similarity scores, then forms a weighted combination of those values.

Example

In translation, attention can connect a generated word to the source words that explain it.

Common Misunderstandings

Attention weights are not always faithful explanations.
Attention is a mechanism, not a complete model architecture by itself.

Same Category

More from Transformers and LLMs

Context WindowA context window is the maximum amount of input and generated tokens a model can consider at one time.Cross-AttentionCross-attention lets one sequence attend to information from a different sequence or modality.Hybrid AttentionHybrid attention combines different attention patterns, such as local and global attention, inside one model.KV CacheA KV cache stores attention keys and values from previous tokens so autoregressive generation does not recompute them every step.Multimodal ModelA multimodal model can process or generate more than one kind of data, such as text, images, audio, or video.Positional EncodingPositional encoding gives a transformer information about token order and position.

Related Reading

Essays that use this concept

Transformers and LLMsWhat Is Attention in Transformers? Explained IntuitivelyTransformers and LLMsSelf-Attention vs Cross-Attention in Transformers

Start here

Need the broader concept map?

Return to the glossary index to move from this definition into adjacent AI, ML, mathematics, and computer science terms.

Browse Glossary