KV Cache
Caching keys and values of processed tokens during inference so history is not recomputed for every new token.
In plain language
Autoregressive generation attends to the full history at every step. KV caching stores keys and values of past tokens so a new token only computes its query and appends to the cache, cutting latency substantially. The cost is memory that grows with context length and concurrency, so long-context serving uses paged caches (PagedAttention), quantized caches or conversation trimming. Cache hit rate directly affects inference cost and time-to-first-token.
Example
Keeping the KV cache across turns avoids recomputing prior context, but memory grows with context length.
Recommended tools
Related terms
Context Window
The maximum number of tokens a model can read and remember at once, including input and output.
Inference
The computation a trained model performs to produce output from an input.
Latency
The time from sending a request to receiving the model response.
Agent Memory
A mechanism for an agent to store and retrieve user preferences, past tasks and key facts across sessions.