Transformers and LLMs

KV Cache, explained

A KV cache stores attention keys and values from previous tokens so autoregressive generation does not recompute them every step.
Back to Glossary

Part of the Semantic Notion technical glossary.

Short Definition

A KV cache stores attention keys and values from previous tokens so autoregressive generation does not recompute them every step.

Intuition

When a model generates one token at a time, it can remember earlier attention data instead of rebuilding it from scratch.

Technical Definition

KV cache is the stored set of key and value tensors for prior sequence positions used during transformer decoding.

Example

A chat model uses a KV cache to speed up each next-token generation step.

Common Misunderstandings

KV cache improves decoding speed but consumes memory.
It does not make the model remember information outside its context window.

Same Category

More from Transformers and LLMs

AttentionAttention is a neural network mechanism that lets a model decide which parts of an input are most relevant when producing a representation or output.Context WindowA context window is the maximum amount of input and generated tokens a model can consider at one time.Cross-AttentionCross-attention lets one sequence attend to information from a different sequence or modality.Hybrid AttentionHybrid attention combines different attention patterns, such as local and global attention, inside one model.Multimodal ModelA multimodal model can process or generate more than one kind of data, such as text, images, audio, or video.Positional EncodingPositional encoding gives a transformer information about token order and position.

Related Reading

Essays that use this concept

Voice AI SystemsLatency in Voice AI Systems: Why Fast Responses Matter

Start here

Need the broader concept map?

Return to the glossary index to move from this definition into adjacent AI, ML, mathematics, and computer science terms.

Browse Glossary