KVCache – Key-Value Cache is a mechanism used to store past Gen AI large language model (LLM) layers’ activations (keys and values) during the inference phase. It allows LLMs to bypass recomputation of these activations, improving performance. The cache serves as a repository to “remember” previous information, the pre-computed key and value pairs, reducing the need to reprocess entire sequences repeatedly. The memory-based KVCache applies to the attention mechanism of a transformer model (LLM). This attention layer computes relationships between input tokens (words or subwords) using three items: queries (Q), keys (K), and values (V). During text generation, the model processes one token at a time, predicting the next token based on all previous ones. Without caching, it would need to recompute the keys and values for all prior tokens at every step, which is an inefficient process.
Keys (K) represent the “context” or features of each token in the sequence. Values (V) hold the actual content or information tied to those tokens.
By maintaining a KVCache, the model only computes K and V for the new token at each step, appending them to the cache, and then uses the full set of cached K-V pairs to compute attention scores with the current query (Q). This drastically reduces computational overhead.
