Transformers and LLMs

Relative Attention, explained

Relative attention adds information about the distance or relationship between tokens directly into attention.
Back to Glossary

Part of the Semantic Notion technical glossary.

Short Definition

Relative attention adds information about the distance or relationship between tokens directly into attention.

Intuition

Many language patterns depend more on how far two tokens are from each other than on their absolute position in a document.

Technical Definition

Relative attention modifies attention scores or representations using learned or computed features of pairwise token positions.

Example

A model can treat nearby modifiers differently from distant context by using relative position signals.

Common Misunderstandings

Relative attention is not simply another name for RoPE.
Relative position methods can be implemented in several different ways.

Same Category

More from Transformers and LLMs

AttentionAttention is a neural network mechanism that lets a model decide which parts of an input are most relevant when producing a representation or output.Context WindowA context window is the maximum amount of input and generated tokens a model can consider at one time.Cross-AttentionCross-attention lets one sequence attend to information from a different sequence or modality.Hybrid AttentionHybrid attention combines different attention patterns, such as local and global attention, inside one model.KV CacheA KV cache stores attention keys and values from previous tokens so autoregressive generation does not recompute them every step.Multimodal ModelA multimodal model can process or generate more than one kind of data, such as text, images, audio, or video.

Related Reading

Essays that use this concept

Transformers and LLMsPositional Encoding ExplainedTransformers and LLMsInkling: What Thinking Machines Actually Shipped

Start here

Need the broader concept map?

Return to the glossary index to move from this definition into adjacent AI, ML, mathematics, and computer science terms.

Browse Glossary