Local AI Glossary

Standardized measurement of a model's capabilities on a defined set of tasks (reasoning, languages, code). Benchmarks let you compare models against each other, but they don't replace testing on your own use case: the benchmark that matters is the one on your questions and your documents.

A serving technique where in-flight requests are grouped into batches dynamically: when one response finishes, its slot in the batch is immediately taken by the next request in the queue, without waiting for all responses in the batch to end. The GPU stays busy almost continuously.

Transmission technique where the server sends the chat not the full answer on every update, but only the new fragment (the delta) relative to the previous one. The result still arrives token by token, but network traffic drops dramatically — up to roughly 99% less in recent implementations — and responses stay fluid even on slow connections.

The time between a request and the model's response. Locally it depends mostly on generation speed (tokens per second), which in turn depends on how large the weights that get re-read every token are: below ~15 tokens/s, interactive use starts to feel slow, whatever the answer quality.

The amount of data RAM can read or write per second (GB/s). It is the dominant factor in CPU decode speed: each token requires re-reading the model weights, so an 8B at Q4 (~5 GB) on 64 GB/s does roughly 11-12 tok/s. It matters less on GPU, because VRAM has far more bandwidth.

A technique where the model is trained to predict multiple future tokens in a single step instead of one. It lets you propose several candidates without extra calls and underpins speculative decoding: the checkpoint ships the MTP modules already trained.

A memory-management technique introduced by vLLM: KV cache memory is allocated in small blocks (pages) rather than one contiguous span, the way an operating system handles virtual memory. It removes fragmentation and wasted memory and lets many concurrent requests share the same GPU.

An inference technique: a small draft model proposes tokens and the large model verifies them in a single pass, accepting or correcting them. The result is more tokens per second with no quality change. Modern variants: MTP (Multi-Token Prediction, built into Qwen3.8 checkpoints) and EAGLE (e.g. on Mistral Medium 3.5).

The number of tokens an inference system generates per unit of time, usually measured in tokens per second. There is overall throughput (all concurrent requests) and single-user latency: two numbers that tell different stories and should be measured separately.