Local AI Glossary

The C/C++ tensor library that llama.cpp is built on (named after GGML, the format that made it famous). It defines tensors, operations and backends — CPU, GPU, Apple Silicon — on which models run in local inference. It is the hardware layer of the inference stack: runtimes like Ollama and llama.cpp sit on top of it.

Attention scheme where several query heads share the same group of Key and Value matrices, reducing the KV-head count compared to MHA. Lowers the cost and memory-access speed during decoding with a minimal quality trade-off.

Family of attention schemes (DeltaNet, Gated DeltaNet, Mamba, etc.) that replace the linearly growing KV cache with a fixed-size recurrent state, drastically reducing memory at long context. It's the technique behind hybrid-attention models like Qwen3.8-27B.

Model architecture in which the model counts many billions of parameters in total, but for each token activates only a small subset (the 'experts' selected by a router). The result is the quality of a large model with the speed and compute cost of a small one. Example: Qwen3 30B-A3B has 30.5B total parameters but uses only 3.3B per token. The price is that the whole model weight must still be loaded into memory.

Retrieval Augmented Generation. Architecture that combines a language model with a document retrieval system. Instead of relying solely on model knowledge, the system retrieves relevant documents from a database before generating the response. This reduces hallucinations and makes the system updatable without retraining.

The stage of a RAG system where, from the user's question, the system searches the vector database for the most similar text blocks and adds them to the prompt. The quality of retrieval is the factor that determines the quality of the final answer.

Feature introduced in Qwen3 where the model can generate a long intermediate reasoning step before producing the final answer. It improves quality on logic, math and planning problems, at the cost of more tokens (and therefore more time and memory) per answer. It can be switched on or off, and in Qwen3 the model also decides for itself whether to reason.

Neural network architecture introduced in 2017 (Attention Is All You Need) that revolutionized NLP. Unlike recurrent networks, Transformers use attention to process the entire input simultaneously. All modern LLMs (GPT, Llama, Mistral, Claude) are based on this architecture.

Apple's technology that fuses two M5 Max dies into a single M5 Ultra chip, presented to the system as one processor. The unified memory of both dies forms a single pool (up to 512GB) accessible to CPU, GPU and Neural Accelerators.

The Apple Silicon architecture in which CPU and GPU share the same pool of RAM. For local AI, the entire model can reside in that memory without copying data between system and video memory.

Voiced Language Model with a Varying-rate adversarial (VITS): a neural text-to-speech architecture that generates the waveform directly from text without a separate vocoder. It underpins Piper and Kokoro-82M.