Short Definition
Latency is the time between a request and the system response.
Intuition
A model can be accurate but still feel unusable if users wait too long for each answer.
Technical Definition
Latency measures end-to-end or component-level response delay, often split into network, retrieval, model inference, tool, and rendering time.
Example
Voice AI systems need low latency so conversations do not feel awkward or interrupted.
Common Misunderstandings
Latency is different from throughput.
Optimizing model speed alone may not fix end-to-end latency.