Transformers and LLMs

Hybrid Attention, explained

Hybrid attention combines different attention patterns, such as local and global attention, inside one model.
Back to Glossary

Part of the Semantic Notion technical glossary.

Short Definition

Hybrid attention combines different attention patterns, such as local and global attention, inside one model.

Intuition

The model uses cheap local attention where nearby context is enough and global attention where full-context access matters.

Technical Definition

Hybrid attention architectures mix attention masks or attention mechanisms across layers to balance compute, memory, and context access.

Example

Inkling alternates sliding-window attention with global attention layers.

Common Misunderstandings

Hybrid attention is a design family, not one fixed algorithm.
It does not remove the need to test long-context behavior empirically.

Same Category

More from Transformers and LLMs

AttentionAttention is a neural network mechanism that lets a model decide which parts of an input are most relevant when producing a representation or output.Context WindowA context window is the maximum amount of input and generated tokens a model can consider at one time.Cross-AttentionCross-attention lets one sequence attend to information from a different sequence or modality.KV CacheA KV cache stores attention keys and values from previous tokens so autoregressive generation does not recompute them every step.Multimodal ModelA multimodal model can process or generate more than one kind of data, such as text, images, audio, or video.Positional EncodingPositional encoding gives a transformer information about token order and position.

Related Reading

Essays that use this concept

Transformers and LLMsInkling: What Thinking Machines Actually Shipped

Start here

Need the broader concept map?

Return to the glossary index to move from this definition into adjacent AI, ML, mathematics, and computer science terms.

Browse Glossary