Access Denied You don’t have permission to access “http://zeenews.india.com/technology/aws-outage-in-uae-amazon-data-centre-hit-by-objects-fire-disrupts-ec2-and-rds-services-as-dubai-abu-dhabi-see-iran-strikes-3022873.html” on this server. Reference #18.eff43717.1772443212.245886df https://errors.edgesuite.net/18.eff43717.1772443212.245886df

In this article, you will learn how key-value (KV) caching eliminates redundant computation in autoregressive transformer inference to dramatically improve generation speed. Topics we will cover include: Why autoregressive generation has quadratic computational complexity How the attention mechanism produces query, key, and value representations How KV caching works in practice, including pseudocode and memory trade-offs Let’s get started. KV Caching in LLMs: A Guide for DevelopersImage by Editor Introduction Language models generate text one token at a time, reprocessing the entire sequence at each step. To generate token n, the model recomputes attention over all (n-1) previous tokens. This creates \( O(n^2) \) complexity, where computation grows quadratically with sequence length, which becomes a major bottleneck for inference speed. Key-value (KV) caching eliminates this redundancy by leveraging the fact that the key and value projections in attention do not change once computed for a token. Instead of recomputing them at each step, we cache and reuse them. In practice, this can reduce redundant computation and provide 3–5× faster inference, depending on model size and hardware. Prerequisites This article assumes you are familiar with the following concepts: Neural networks and backpropagation The transformer architecture The self-attention mechanism in transformers Matrix multiplication concepts such as dot products, transposes, and basic linear algebra If any of these feel unfamiliar, the resources below are good starting points before reading on. The Illustrated Transformer by Jay Alammar is one of the clearest visual introductions to transformers and attention available. Andrej Karpathy’s Let’s Build GPT walks through building a transformer from scratch in code. Both will give you a solid foundation to get the most out of this article. That said, this article is written to be as self-contained as possible, and many concepts will become clearer in context as you go. The Computational Problem in Autoregressive Generation Large language models use autoregressive generation — producing one token at a time — where each token depends on all previous tokens. Let’s use a simple example. Start with the input word: “Python”. Suppose the model generates: Input: “Python” Step 1: “is” Step 2: “a” Step 3: “programming” Step 4: “language” Step 5: “used” Step 6: “for” … Input: “Python” Step 1: “is” Step 2: “a” Step 3: “programming” Step 4: “language” Step 5: “used” Step 6: “for” … Here is the computational problem: to generate “programming” (token 3), the model processes “Python is a”. To generate “language” (token 4), it processes “Python is a programming”. Every new token requires reprocessing all previous tokens. Here is a breakdown of tokens that get reprocessed repeatedly: “Python” gets processed 6 times (once for each subsequent token) “is” gets processed 5 times “a” gets processed 4 times “programming” gets processed 3 times The token “Python” never changes, yet we recompute its internal representations over and over. In general, the process looks like this: Generate token 1: Process 1 position Generate token 2: Process 2 positions Generate token 3: Process 3 positions … Generate token n: Process n positions Generate token 1: Process 1 position Generate token 2: Process 2 positions Generate token 3: Process 3 positions … Generate token n: Process n positions This gives us the following complexity for generating n tokens:\[\text{Cost} = 1 + 2 + 3 + \cdots + n = \frac{n(n+1)}{2} \approx O(n^2)\] Understanding the Attention Mechanism and KV Caching Think of attention as the model deciding which words to focus on. The self-attention mechanism at the core of transformers computes: \[\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V\] The mechanism creates three representations for each token: Query (Q): Each token uses its query to search the sequence for relevant context needed to be interpreted correctly. Key (K): Each token broadcasts its key so other queries can decide how relevant it is to what they are looking for. Value (V): Once a query matches a key, the value is what actually gets retrieved and used in the output. Each token enters the attention layer as a \( d_{\text{model}} \)-dimensional vector. The projection matrices \( W_Q \), \( W_K \), and \( W_V \) — learned during training through backpropagation — map it to \( d_k \) per head, where \( d_k = d_{\text{model}} / \text{num\_heads} \). During training, the full sequence is processed at once, so Q, K, and V all have shape [seq_len, d_k], and \( QK^T \) produces a full [seq_len, seq_len] matrix with every token attending to every other token simultaneously. At inference, something more interesting happens. When generating token \( t \), only Q changes. The K and V for all previous tokens \( 1 \ldots t-1 \) are identical to what they were in the previous step. Therefore, it is possible to cache these key (K) and value (V) matrices and reuse them in subsequent steps. Hence the name KV caching. Q has shape [1, d_k] since only the current token is passed in, while K and V have shape [seq_len, d_k] and [seq_len, d_v], respectively, growing by one row each step as the new token’s K and V are appended. With these shapes in mind, here is what the formula computes: \( QK^T \) computes a dot product between the current token’s query and every cached key, producing a [1, seq_len] similarity score across the full history. \( 1/\sqrt{d_k} \) scales scores down to prevent dot products from growing too large and saturating the softmax. \( \text{softmax}(\cdot) \) converts the scaled scores into a probability distribution that sums to 1. Multiplying by V weights the value vectors by those probabilities to produce the final output. Comparing Token Generation With and Without KV Caching Let’s trace through our example with concrete numbers. We will use \( d_{\text{model}} = 4 \). Real models, however, typically use 768–4096 dimensions. Input: “Python” (1 token). Suppose the language model generates: “is a programming language”. Without KV Caching At each step, K and V are recomputed for every token in the sequence, and the cost grows as each token is added. Step Sequence K & V Computed 0 Python

Access Denied You don’t have permission to access “http://zeenews.india.com/technology/4g-vs-5g-vs-lte-which-network-mode-is-best-for-faster-internet-and-better-battery-life-3022562.html” on this server. Reference #18.eff43717.1772369043.1687ffd4 https://errors.edgesuite.net/18.eff43717.1772369043.1687ffd4

Access Denied You don’t have permission to access “http://zeenews.india.com/technology/us-israel-strike-on-iran-hits-flights-is-flightradar24-down-here-s-how-to-track-real-time-flight-status-on-live-global-map-3022212.html” on this server. Reference #18.c4f43717.1772293925.b179fe5 https://errors.edgesuite.net/18.c4f43717.1772293925.b179fe5

Access Denied You don’t have permission to access “http://zeenews.india.com/technology/whatsapp-to-roll-out-status-ads-promoted-channels-globally-users-can-skip-or-block-ads-chats-remain-end-to-end-encrypted-3022001.html” on this server. Reference #18.8e277368.1772263164.aa2ac8 https://errors.edgesuite.net/18.8e277368.1772263164.aa2ac8

Using large language models (LLMs) — or their outputs, for that matter — for all kinds of machine learning-driven tasks, including predictive ones that were already being solved long before language models emerged, has become something of a trend.

Access Denied You don’t have permission to access “http://zeenews.india.com/technology/chatgpt-naughty-chat-feature-adults-can-talk-on-spicy-topics-through-erotica-mode-update-likely-soon-3021912.html” on this server. Reference #18.c4f43717.1772230113.17202d3 https://errors.edgesuite.net/18.c4f43717.1772230113.17202d3

Access Denied You don’t have permission to access “http://zeenews.india.com/technology/is-whatsapp-web-down-today-5-easy-fixes-you-can-try-in-this-situation-3021836.html” on this server. Reference #18.26f63717.1772196088.15c848e8 https://errors.edgesuite.net/18.26f63717.1772196088.15c848e8

Access Denied You don’t have permission to access “http://zeenews.india.com/technology/91-per-cent-indian-firms-now-pilot-workplace-ai-report-3021077.html” on this server. Reference #18.d42fdf17.1772169385.d277d5 https://errors.edgesuite.net/18.d42fdf17.1772169385.d277d5

Access Denied You don’t have permission to access “http://zeenews.india.com/technology/social-media-drives-77-pc-of-retail-discovery-led-by-meta-platforms-report-3021163.html” on this server. Reference #18.54fdd417.1772143129.193ce235 https://errors.edgesuite.net/18.54fdd417.1772143129.193ce235