GPU or CPU: when the CPU is enough

When local AI comes up, the reflex is always the same: "how much VRAM do I need?" In many cases, though, that's the wrong question. The answer depends on what you want to do, and for a growing number of scenarios the CPU is entirely sufficient — even on hardware you would never have considered.

This article clears up the trade-off: what actually separates CPU from GPU, when you can skip the GPU, and how to measure the numbers yourself on your own hardware.

How inference works: two phases, one bottleneck

Every request to an LLM goes through two phases:

1. Prefill: the model reads the prompt (the input tokens) in one shot. This is the phase that determines how slow the first token is — the so-called time-to-first-token.
2. Decode: the model generates the response, one token at a time. This is the phase that determines how fast the text "streams" (the famous tokens per second).

The two phases have different requirements, and that's where the whole CPU vs GPU question lives.

Prefill: raw compute wins

Prefill is a pure raw compute problem: huge matrix multiplications that a GPU executes in parallel across thousands of cores. A dedicated GPU does the prefill up to an order of magnitude faster than a CPU: on a long prompt, the first token arrives in a flash with a GPU, while it can take a few seconds with a CPU.

Decode: memory bandwidth wins

Token-by-token generation is a different problem: for every token, the model has to re-read all the weights (and the KV cache) from memory. Generation speed depends almost entirely on memory bandwidth — how many bytes per second you can read.

And this is where the CPU holds its own:

• dual or quad channel DDR5 memory reaches 50–100 GB/s;
• a consumer GPU's VRAM reaches 300–1000 GB/s;
• Apple Silicon's unified memory reaches over 800 GB/s.

In practice, for small and mid-sized models (1 to about 8 billion parameters, Q4 quantization), a modern multi-channel CPU generates tokens at perfectly usable speeds: 5–20 tokens per second depending on the model. That's the pace of a person reading aloud: plenty for conversations, summaries and short tasks.

When the CPU is enough

1. Small models (0.5B–4B)

Below 4 billion quantized parameters, the weights take 0.5–3 GB of RAM: they fit in the processor's L3 cache and memory bandwidth is not even a limit. On a 5–6 year old laptop with 12 GB of RAM, a 1B model generates over 20 tokens per second.

2. Interactive use with short answers

Everyday chat, rephrasing, classification, data extraction: short answers don't suffer from the slow prefill, and generation on CPU is smooth.

3. Batch and overnight workloads

If the model has to process documents, label data or generate reports while you sleep, speed is not the goal: cost is. A CPU draws 50–150 W against 300–600 W for a gaming GPU, and produces no heat or noise.

4. Hardware you already own

If your laptop has no discrete GPU (and the vast majority of consumer laptops don't), local AI is still accessible. There is nothing to buy: you need RAM.

5. Raspberry Pi and edge

On a Raspberry Pi 5, a 0.5–1B model generates 5–10 tokens per second: too little for a chat, perfect for automation, voice triggers and offline assistants that respond to commands.

6. Reliability first

In a production context (a home server, an automation), the CPU is the most stable solution: no drivers to update, no VRAM "out of memory", predictable power consumption.

When you need a GPU

• Large models (13B, 30B, 70B and up): system RAM cannot keep the pace — 70B at Q4 needs about 40 GB and drops to 1–2 tokens per second on CPU. With 24 GB of VRAM (RTX 3090/4090), tokens arrive at 20–30 per second.
• Long prompts or large context: a prefill of 100k tokens on CPU can take tens of seconds.
• Many users in parallel: per-user throughput on CPU collapses as soon as two or three requests overlap.
• Critical time-to-first-token: real-time assistants, tools that wait on the first response.

Real numbers: what to expect

Reference test on a mid-range Intel i5 laptop (12 GB RAM, no useful GPU), with Ollama and Q4_K_M quantized models:

Model	File size	Generation speed
Qwen3 0.6B	~0.5 GB	34–36 tok/s
TinyLlama 1.1B	~0.6 GB	25–28 tok/s
Gemma 3 1B	~0.8 GB	~18 tok/s
Gemma 4 E2B	~7 GB	~10 tok/s
Granite 4 3B	~2 GB	8–9 tok/s
Phi-4 Mini 3.8B	~2.5 GB	~7 tok/s
OpenHermes 7B	~4 GB	4 tok/s
Ministral 3 8B	~6 GB	3 tok/s

The rule of thumb is simple: below ~10 tokens per second the conversation starts to feel heavy; above 20 it feels fluid. And the numbers scale with memory bandwidth: a dual-socket workstation with 12 DDR5 channels (or a Mac with unified memory) generates 5–10× more tokens per second on the same model than a laptop.

Measure your hardware in 5 minutes

With Ollama:

ollama run qwen3:4b --verbose

The --verbose flag shows eval rate (generation tokens/s) and prompt eval rate (prefill tokens/s).

With llama.cpp, to force the CPU (zero layers on the GPU):

llama-server -m qwen3-4b-q4_k_m.gguf -ngl 0 -t 8 --port 8080

Then query the API:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3","messages":[{"role":"user","content":"Tell me a joke"}]}'

The two numbers to note: eval rate (response speed) and time to first token (prefill speed). If the eval rate is above 10 tokens per second and your use case has no huge prompts, the CPU is enough for you.

Practical rules to squeeze the CPU

1. RAM first: weights + context must fit in RAM. As a rule of thumb, 1 GB per billion parameters at Q4, plus 0.5–1 GB headroom.
2. Threads = physical cores, not total cores. An 8-core/16-thread CPU runs better with -t 8 than with -t 16.
3. Q4_K_M quantization is the best compromise: nearly Q8 quality, double the speed.
4. Moderate context: the KV cache grows with context. If you don't need 32k tokens, don't allocate them.
5. Dual channel is gold: two RAM sticks instead of one doubles the bandwidth, and with it the tokens per second.
6. On multi-socket systems, pin the threads to one NUMA node: reading RAM from the other socket costs up to 30% of speed.

The Apple Silicon case

A modern Mac is the perfect counter-example: the integrated GPU is fast, but the memory is unified — CPU and GPU read the same bytes. A model that "doesn't fit in VRAM" fits fine in RAM on a Mac with 32–64 GB, and runs at GPU speeds. If you have a Mac, the "CPU or GPU?" question becomes moot: measure, and you'll find the model is faster than you expected.

In summary

Scenario	The right choice
Chat with 0.5–4B models	CPU, without thinking twice
Batch automation	CPU (cost + silence)
Laptop without GPU	CPU (the only option — and it works)
Raspberry Pi / edge	CPU
13B+ models in quality	GPU (or a Mac with unified memory)
Long prompts, many users	GPU

A GPU is not the ticket into local AI: it's an optional upgrade that matters when quality or speed demands it. Your current hardware, with the right models, is probably already enough. Try the numbers on your machine, then decide.

Article generated with AI