KV cache: why your RAM needs more than just the model

You downloaded an 8B model in Q4, loaded it into Ollama or llama.cpp, and the program tells you the memory footprint is far bigger than the model's weights. That's not a bug: it's the KV cache (key-value cache). Today we'll see what it is, how much it takes, and how to manage it — with the real numbers of a model you can run on a laptop.

If you want the "why" of the model's own compression — what quantization does and where it loses quality — the article on Quantization covers that ground. Here we're talking about something different: the memory the model uses while it generates, token by token.

What the KV cache is

A transformer model reads text in "heads" (tokens) and, for each new token, looks at the previous ones through attention. To do that, it computes three matrices per token: Query (Q), Key (K) and Value (V). If it started from scratch on every new token, it would recompute K and V for every token it had already read: the cost would grow quadratically and generation would be pointlessly slow.

The KV cache is the shortcut: the K and V values for each token, once computed, are stored in memory. When the next token arrives, the model reads the cache instead of recomputing. It's the conversation's short-term memory: the longer the context, the more space it takes.

How much space: the formula

Its size depends on four parameters you find in the model's config file (the config.json on Hugging Face) and on the precision it's stored at:

bytes per token = 2 × layers × KV_heads × head_dim × bytes per value

The "2" is for the two matrices, K and V. "KV_heads" is the number of heads the cache uses: in models with GQA (Grouped-Query Attention) it differs from the number of query heads, and is usually smaller — but for total memory it's always the KV-head count that matters. "head_dim" is the size of each head. "bytes per value" is 2 for fp16, 1 for int8.

Let's run the numbers for real. From Llama 3.1 8B's config.json: 32 layers, 8 KV heads, head_dim 128, fp16.

2 × 32 × 8 × 128 × 2 = 131 072 bytes = 128 KiB per token

Same model, int8 for the KV cache: 65 536 bytes, i.e. 64 KiB per token — half.

To compare a different model, Qwen3 8B's config.json: 36 layers, 8 KV heads, head_dim 128.

2 × 36 × 8 × 128 × 2 = 147 456 bytes = 144 KiB per token (fp16)

A worked example: 8B at 8K or 128K context

Now we multiply by the context length, which is the main multiplier.

Llama 3.1 8B (128 KiB per token in fp16):

Context	Tokens	KV cache
8K	8 192	≈ 1.0 GiB
32K	32 768	≈ 4.0 GiB
128K	131 072	≈ 16 GiB

Qwen3 8B (144 KiB per token in fp16):

Context	Tokens	KV cache
8K	8 192	≈ 1.1 GiB
32K	32 768	≈ 4.5 GiB
128K	131 072	≈ 18 GiB

Here's the part that surprises people: an 8B model that weighs 5–6 GB on disk in Q4 can demand 20 GB of total memory if you run it at 128K context in fp16. At long context, the KV cache outweighs the model itself.

That's why, when you size the memory a setup needs, you have to add up weights + KV cache + activations. The weights are a constant (they depend on the model and its quantization); the KV cache is variable (it depends on context and its precision). To understand how to read the VRAM a model requires, the article Hardware for local AI starts there.

How you configure it

You have three main levers.

1. The context length. The most obvious. In Ollama you set it with num_ctx in the Modell file, in LM Studio it's the "Context length" slider, in llama.cpp it's the -c flag (for example -c 8192). If you're not running very long conversations or loading whole documents, 8K or 16K is often more than enough, and the KV cache stays under 2 GiB.

2. Quantizing the KV cache. The most effective lever. In llama.cpp, the --cache-type-k and --cache-type-v flags let you store the K and V matrices in int8 (Q8_0) instead of fp16. Half the memory, with a negligible quality loss for most models. Ollama and LM Studio have equivalent options in their settings panel. The result: a 128K context that took 16 GiB in fp16 drops to 8 GiB in Q8_0 on Llama 3.1 8B.

3. Flash Attention. It doesn't shrink the KV cache, but it makes the attention computation more memory-peak-efficient: instead of materializing the full attention matrix, Flash Attention processes it in blocks. In llama.cpp you turn it on with --flash-attn (or -fa). The llama.cpp documentation describes it as an optimization to keep on whenever the backend supports it.

GQA: why newer models take less

Older models (GPT-3, Llama 2 7B) used MHA: every query head had its own K and V, so the KV-head count equaled the query-head count (for example 32). Models with GQA (Llama 3, Qwen3, Mistral) group several query heads under a single K-V group, cutting the KV-head count to 8 or 12. That doesn't change the total memory per token (the formula is the same), but it makes decoding faster because the model reads fewer separate tensors from memory at every step.

What to take away

The KV cache is not a side detail: it's the part of the memory you control, because it depends on the context you choose. The model's weights are fixed; the KV cache is variable. When a model doesn't fit in your RAM, the first place to look is the context, not the model's weight — often just dropping it from 128K to 32K, or quantizing the KV cache to int8, makes the problem disappear.

The 8B model on your laptop takes 6 GB of weights. But the RAM it needs to run well depends on the context you give it — and that's the part you can tune yourself, without changing hardware.

Article generated with AI