Local AI Glossary

Small file of trained weights (low-rank matrices) added to the base model without modifying it: a few MB in size, loadable and removable at will, so the same model can carry several adapters for different behaviors.

A program that uses a language model to reach a goal in multiple steps: it decides which tools to call (search, calculator, terminal, browser), reads the results, and decides the next step until the task is done or fails. The difference from a chat: the agent acts on the environment, and its behavior depends on the harness — the code around it — that assembles observations and actions.

Apple's family of chips (M1, M2, M3, M4 and later) that has equipped Macs since 2020. For local AI the key feature is the unified memory shared between CPU and GPU: a 32 GB Mac can run models that on a PC would require a dedicated GPU.

Standardized measurement of a model's capabilities on a defined set of tasks (reasoning, languages, code). Benchmarks let you compare models against each other, but they don't replace testing on your own use case: the benchmark that matters is the one on your questions and your documents.

The usage threshold (tokens per month, requests, work hours) above which the monthly cost of local compute — hardware amortization + electricity — falls below the cost of the equivalent cloud API. Below it the API wins; above it, local wins.

The splitting of a document into small text blocks before indexing. Each block becomes a searchable entry in the vector database: block size is one of the first parameters to tune when RAG results are not convincing.

A single shared memory pool across CPU and GPUs where all cores read and write the same memory without copies. This is the model of NVIDIA's Grace-based systems (e.g. the DGX Station's 748GB) and of Apple's unified memory: the LLM lives in one place, with no copies between RAM and VRAM.

The maximum number of tokens a model can process simultaneously in a single interaction. A larger context window allows the model to read longer documents, more extensive conversations, and more complex code. Modern models support 8K to 128K+ context tokens.

A serving technique where in-flight requests are grouped into batches dynamically: when one response finishes, its slot in the batch is immediately taken by the next request in the queue, without waiting for all responses in the batch to end. The GPU stays busy almost continuously.

Customer Relationship Management: software that manages customer relationships — contacts, quotes, negotiations, history. The quality of CRM data determines how much a business can automate sales and follow-up.

Compute Unified Device Architecture: NVIDIA's GPU programming stack — libraries, compiler and runtime that run model inference on NVIDIA GPUs. It is the default backend for Ollama and llama.cpp on NVIDIA hardware: optimizations (Flash Attention, quantized kernels, FP4 formats) arrive on CUDA first.

A model that, given a state and a typed question (yes/no, choice among options, score), returns a probability distribution over the options in a single pass, without generating tokens. It is the "System 1" of AI: fast and calibrated, used for routing, classification and quality control, in place of a generative LLM when you don't need to write.

Small, fast AI models designed to answer multiple-choice questions with a structured label and a probability per option, instead of generating free text. The name comes from Jev, the acronym identifying this class of models. They are built for low-latency triage, routing, classification and moderation: where a large LLM would generate a paragraph, a decision model returns the option and its confidence level.

The second phase of inference: the model generates the response token by token. Each token requires re-reading all model weights and the KV cache from memory, so the speed (tokens per second) is driven mostly by memory bandwidth.

Transmission technique where the server sends the chat not the full answer on every update, but only the new fragment (the delta) relative to the previous one. The result still arrives token by token, but network traffic drops dramatically — up to roughly 99% less in recent implementations — and responses stay fluid even on slow connections.

A model in which every token passes through all parameters: there is no expert selection, the full capacity is active at each step. Heavier to decode than a same-total-size MoE, but without expert routing. Example: Qwen3.8-27B.

Architecture that applies Transformers to diffusion models for image generation. Instead of the classic convolution it uses attention to turn noise into an image. The generation component of Qwen-Image is a 32-layer DiT.

Family of speculative decoding techniques (EAGLE paper, arXiv:2401.15077): a small draft model reads the big model's hidden states and predicts the next tokens from there. Sampling stays lossless, like classical speculative sampling. Official EAGLE checkpoints (for example Mistral-Medium-3.5-128B-EAGLE) are served natively by vLLM and SGLang.

Vector representation of text. A model transforms words, sentences, or documents into an array of numbers (vector) that captures the semantic meaning. Embeddings are used for semantic search, text similarity, and as a foundation for RAG systems.

MoE-specific technique: instead of splitting every layer, the experts (conditional sub-networks) are distributed across GPUs. A token only invokes the active experts, so VRAM is used more efficiently than with tensor parallelism.

Additional training process on a pre-trained model, using a specific dataset to specialize it for a particular task. For example, you can fine-tune Llama 7B with legal documents to create a model expert in law. Popular techniques include LoRA and QLoRA.

Technique that reduces memory access during attention by splitting it into blocks and keeping intermediate results in fast memory. It lowers bandwidth demand and speeds up prefill and decode, especially on long contexts. In llama.cpp it is enabled with the -fa flag.

8-bit floating-point format for large-model weights and activations (E4M3 and E5M2 variants). Half the VRAM of FP16 at minimal quality loss on weights: recent models (e.g. Aleph Alpha's Kolibri) ship natively in FP8. Unlike NVFP4 or MXFP4, it is natively supported by base libraries (cuBLAS) without per-block micro-scaling.

The largest-scale language models (OpenAI's GPT series, Anthropic's Claude, etc.) which today are almost always available only via paid APIs: the lowest per-token prices, but with provider dependency and rate limits.

The current generation of graphics memory: up to 32 Gbps per pin — 1 TB/s on a 256-bit bus (RTX 5090). For local inference, memory bandwidth sets generation speed: NVIDIA's Blackwell cards use it across the 50 series, and the 2026 GDDR7 price surge has pushed street prices up across the mid-range.

Google DeepMind's open language model family, built on the same technology as the Gemini series. Sizes range from 1B to 27B parameters; Gemma 3 adds a 128K-token context, 140+ languages and multimedia input, while Gemma 4 (2026) moves to the Apache 2.0 license.

The C/C++ tensor library that llama.cpp is built on (named after GGML, the format that made it famous). It defines tensors, operations and backends — CPU, GPU, Apple Silicon — on which models run in local inference. It is the hardware layer of the inference stack: runtimes like Ollama and llama.cpp sit on top of it.

GGML Unified Format. The standard file format for distributing quantized LLM models, especially with llama.cpp. It supports storing the model, quantized weights, and tokens. It is the predominant format for local inference on consumer hardware.

Graphics processing unit. For local AI the GPU is the component that accelerates inference: its video memory (VRAM) determines how large a model you can run. Apple's GPUs, built into Apple Silicon, run on unified memory instead of separate VRAM.

Attention scheme where several query heads share the same group of Key and Value matrices, reducing the KV-head count compared to MHA. Lowers the cost and memory-access speed during decoding with a minimal quality trade-off.

Phenomenon where an LLM generates false or invented information with high confidence. Hallucinations are a well-known issue with language models: they can invent facts, quotes, or non-existent references. RAG and fine-tuning are techniques to reduce them.

The leading open-source platform for AI models, datasets, and applications. The Hugging Face Hub hosts thousands of open-source LLM models (Llama, Mistral, Phi, and many others) ready for download and local use. It is also an organization that develops libraries such as Transformers and Datasets.

The process of using a trained model to generate responses or make predictions on new data. In local AI, inference runs on your computer instead of remote servers. This ensures privacy and consistent response times without relying on internet connections.

A version of a model adapted for conversation: it understands instructions and answers directly, recognizable by the 'instruct' or '-it' suffix. The base model, without this training, is a raw text completer used for fine-tuning.

Key-Value cache. The memory a transformer model uses to remember the conversation: every new token adds a small entry that grows with context length and the number of users served in parallel. It is the VRAM component that varies with your settings and must be accounted for per user.

The time between a request and the model's response. Locally it depends mostly on generation speed (tokens per second), which in turn depends on how large the weights that get re-read every token are: below ~15 tokens/s, interactive use starts to feel slow, whatever the answer quality.

Family of attention schemes (DeltaNet, Gated DeltaNet, Mamba, etc.) that replace the linearly growing KV cache with a fixed-size recurrent state, drastically reducing memory at long context. It's the technique behind hybrid-attention models like Qwen3.8-27B.

Family of open-source LLM models by Meta. Llama 3 is the current generation, with 8B and 70B parameter versions. Llama is one of the most widely used models for local AI, with broad compatibility with Ollama, llama.cpp, and most inference frameworks.

Library written in C++ to run LLM models on consumer CPUs and GPUs. It is the underlying engine for many local inference tools, including Ollama. It supports GGUF quantization and can run large models on limited hardware by leveraging both CPU and GPU.

Large Language Model. An AI model trained on massive amounts of text, capable of understanding and generating human language. LLMs like Llama, Mistral, and GPT power chatbots, virtual assistants, and automated writing tools.

The network interface of your own computer, reachable at address 127.0.0.1 (IPv4) or ::1 (IPv6). Services running locally — the Ollama server, LM Studio, web UIs — listen there: your applications connect to the local model without traversing the external network. A well-configured firewall that blocks external access to localhost is the first line of defense: no program can send your content outside without an outbound connection from the machine showing it.

Low-Rank Adaptation. An efficient fine-tuning technique that updates only a small fraction of the model weights instead of all parameters. LoRA enables fine-tuning a 7B model on a single consumer GPU, generating adaptation files of a few MB instead of gigabytes.

An attack in which a malicious system inserts itself between two parties that believe they are communicating directly. Sender and recipient think they are talking to each other, but a third party sees (and sometimes modifies) what passes. The main remedy is TLS/HTTPS with validated certificates: it makes the communication encrypted and authenticated, so a man in the middle cannot read or alter the content.

Memory bandwidth (GB/s). How fast the GPU reads the model's weights from VRAM. It is the factor that limits token generation speed: for the same model, a card with higher bandwidth produces more tokens per second. Measured in GB per second and listed in every card's official specs.

The amount of data RAM can read or write per second (GB/s). It is the dominant factor in CPU decode speed: each token requires re-reading the model weights, so an 8B at Q4 (~5 GB) on 64 GB/s does roughly 11-12 tok/s. It matters less on GPU, because VRAM has far more bandwidth.

Family of open-source LLM models developed by Mistral AI, a French company. Mistral 7B and Mixtral are among the most popular models for local inference due to their quality/performance ratio. They are available on Hugging Face and supported by Ollama, llama.cpp, and other tools.

Model architecture in which the model counts many billions of parameters in total, but for each token activates only a small subset (the 'experts' selected by a router). The result is the quality of a large model with the speed and compute cost of a small one. Example: Qwen3 30B-A3B has 30.5B total parameters but uses only 3.3B per token. The price is that the whole model weight must still be loaded into memory.

Apple's machine learning framework designed for Apple Silicon chips. It powers many local AI applications on Mac, leveraging unified memory to accelerate inference.

Memory mapping: the mechanism by which the operating system maps a file into a program's memory, reading only the requested parts on demand. It's why models in GGUF format, with their aligned tensors, load in a few seconds: the whole file doesn't have to be copied into RAM.

A technique where the model is trained to predict multiple future tokens in a single step instead of one. It lets you propose several candidates without extra calls and underpins speculative decoding: the checkpoint ships the MTP modules already trained.

Microscaling FP4: a 4-bit floating-point format where each block of 32 weights shares one scaling exponent. It is the native format of gpt-oss: weights ship pre-quantized, with no conversion step. Compared to Q4_K_M GGUF, MXFP4 is optimized for Blackwell tensor cores and NVIDIA hardware in particular.

Computational system inspired by the human brain, composed of layers of artificial neurons. LLMs are based on the Transformer architecture, a specific family of neural networks optimized for language. Each model is defined by its architecture, weights, and the dataset it was trained on.

Non-Uniform Memory Access: the architecture of multi-socket systems where each CPU accesses its local memory faster than the other socket's memory. Inference servers should pin threads to a single NUMA node, otherwise performance drops by up to 30%.

NVIDIA's 4-bit quantization format (NVIDIA Floating Point 4): represents model weights with 4 bits plus a per-group scale factor, cutting memory by roughly a quarter compared to 16 bits. It is the native format of Blackwell tensor cores and lets 35B models run on a 32GB GPU.

Optical Character Recognition: the technique that converts images and scans into selectable text. For documents not "born digital", it's the step before the language model: without OCR, the model has no text to work on.

Open-source tool to download, manage, and run LLM models locally. Supports Llama, Mistral, Phi, and many more. With a single command, you can start a model and access it via a local REST API. It is one of the easiest ways to get started with local AI.

Software whose source code is public and can be studied, modified, and redistributed. In AI, open-source models like Llama, Mistral, and Gemma can be downloaded and run locally, enabling full control, privacy, and customization.

A model is open-weight when its weights are downloadable and its use and redistribution are governed by a license. Unlike open source, the code, data, and training method are not necessarily public. Most modern models are open-weight.

A memory-management technique introduced by vLLM: KV cache memory is allocated in small blocks (pages) rather than one contiguous span, the way an operating system handles virtual memory. It removes fragmentation and wasted memory and lets many concurrent requests share the same GPU.

Numerical value inside an AI model that is learned during training. A 7B (billion) parameter model has 7 billion values that determine its behavior. More parameters generally mean greater capabilities, but require more computational resources.

Parameter-Efficient Fine-Tuning. Family of techniques that train a small fraction of the model's parameters (LoRA, adapters, learnable prompts) instead of all weights, cutting training memory and time.

The first phase of inference, where the model processes the input prompt (all tokens) in one pass. It determines time-to-first-token: near-instant on a GPU, it can take several seconds on a CPU for long prompts.

The unit cost you pay for inference on APIs and providers: a price per million input tokens and, separately, per million output tokens. Comparing it with your own hardware's per-token cost (electricity bill + amortization) tells you when owning beats calling an API.

The art of writing effective instructions to get better results from a language model. Includes techniques like few-shot prompting (providing examples in the prompt), chain-of-thought (asking the model to reason step by step) and role prompting (defining a role for the model).

Quantized Low-Rank Adaptation. A LoRA variant that quantizes the base model to 4 bits (NF4) before training: it lets you adapt models up to 65B on a single 48 GB GPU, with adapters of a few MB.

A technique that reduces the numerical precision of a model to decrease its size and speed up inference. 4-bit quantization can shrink a model from 40 GB to about 10 GB while retaining most performance. GGUF and GPTQ are standard formats for quantized models.

Training that simulates quantization during training, so the model learns to work with low-precision weights. Unlike post-training quantization (applied cold, e.g. Q4_K_M), QAT preserves much more quality at the same bit width. Google released official QAT checkpoints for the Gemma 4 family in June 2026.

Retrieval Augmented Generation. Architecture that combines a language model with a document retrieval system. Instead of relying solely on model knowledge, the system retrieves relevant documents from a database before generating the response. This reduces hallucinations and makes the system updatable without retraining.

The computer's main memory, used by the CPU for programs and — without a GPU — for loading model weights. In CPU inference, the amount of RAM determines which models you can load and its bandwidth determines the generation speed.

Remote Direct Memory Access: lets one machine read and write directly into another's memory without routing data through their CPUs. On Mac Studio it is used to cluster up to 4 units over Thunderbolt 5, speeding up distributed inference by up to 3x.

Radeon DNA: AMD's consumer GPU architecture. RDNA 3 (RX 7000 series) and RDNA 4 (RX 9000 series) are the families officially supported by ROCm 7 on Linux for local AI, with Ollama and llama.cpp and no workarounds: the GPU targets are gfx1100 for RDNA 3 and gfx1200/gfx1201 for RDNA 4.

The stage of a RAG system where, from the user's question, the system searches the vector database for the most similar text blocks and adds them to the prompt. The quality of retrieval is the factor that determines the quality of the final answer.

AMD's GPU compute software stack: the CUDA equivalent for Radeon hardware. Since ROCm 7, consumer RDNA 3 and 4 cards (RX 7900 XTX, RX 9070 XT) are officially supported on Linux with Ollama and llama.cpp, no workaround environment variables. The alternative path is Vulkan: less optimized but more cross-vendor, and it is the maintained path for Intel Arc cards as well.

The closeness between two texts' vectors based on meaning, not on literal match. It is the principle behind search in a vector database: "brake" and "braking system" are not the same word, but their vectors are neighbors.

A cloud service that runs code or models on GPUs only when there is demand, starting instances in seconds and dropping cost to zero when idle. You pay for actual compute seconds, not for an always-on server.

A family of CPU instructions that process multiple data elements in one operation (AVX, AVX-512, NEON, AMX). This is why modern CPUs can run quantized models at usable speeds: the same operation applies to 8 or 16 numbers at once.

An inference technique: a small draft model proposes tokens and the large model verifies them in a single pass, accepting or correcting them. The result is more tokens per second with no quality change. Modern variants: MTP (Multi-Token Prediction, built into Qwen3.8 checkpoints) and EAGLE (e.g. on Mistral Medium 3.5).

The technology that converts audio into text (speech recognition, ASR). Locally, with models like Whisper Large-v3-Turbo, transcription happens on your own hardware: the audio never leaves the machine, making it the reference option for sensitive documents. Quality is measured by WER (word error rate): turbo sits around 2.5% on clean English.

The set of instructions that defines a model's permanent behavior: tone, role, response format, constraints. Unlike the user prompt (a single question), the system prompt governs every response of the session and is not shown to the model as a request.

The data a program automatically sends to report what it did: version, error type, usage metrics. Telemetry is not the same thing as content: a program can send telemetry without ever touching your prompts or your documents. When you read a privacy policy, the question to ask is what data goes out and under what conditions.

A multidimensional array of numbers: the matrix that makes up each layer of a neural model (weights, biases, embeddings). In a GGUF file every tensor has a name, dimensions, type, and offset; it is the alignment and type of each tensor that determine how the model is loaded and how much memory it takes.

Serving technique that splits each matrix (tensor) of a model across multiple GPUs: each card computes a part of the same layer and the results are recomposed. Used when a single layer doesn't fit on one GPU, e.g. for large MoE models.

Weight representation restricted to {−1, 0, +1} values, often with a shared scale per group. Brings the cost to ~1.58 bits per weight (plus the scale), enabling tens-of-billions-of-parameters models in a few GB, as in Bonsai-2 27B.

Speech synthesis: generating spoken audio from text. Local systems like Piper (VITS/ONNX-based) run in real time on CPU with no GPU: ideal for accessibility pipelines, voice commands and assistants. The trade-off is naturalness versus latency and resources.

Feature introduced in Qwen3 where the model can generate a long intermediate reasoning step before producing the final answer. It improves quality on logic, math and planning problems, at the cost of more tokens (and therefore more time and memory) per answer. It can be switched on or off, and in Qwen3 the model also decides for itself whether to reason.

The number of tokens an inference system generates per unit of time, usually measured in tokens per second. There is overall throughput (all concurrent requests) and single-user latency: two numbers that tell different stories and should be measured separately.

The basic unit of text processed by an LLM. A token is typically a word or part of a word. In English, roughly 1 token ≈ 4 characters. Models have a maximum context measured in tokens: GPT-4 supports 128K tokens, while Llama 3 supports 128K. More tokens = more context but more resources needed.

The component that splits text into tokens, the units the model actually reads. A multilingual tokenizer encodes Italian in fewer tokens than an English-only one, reducing compute and context consumption.

Mechanism where the model, before executing a tool call (sending an email, modifying a file, querying an API), stops and presents the request to the user, who can allow or deny it. It is the most direct way to keep control over what your AI can do with your data: every potentially irreversible action passes through an explicit yes.

A language model's ability to call external functions (APIs, calculator, database, browser) while answering: the model produces a structured call with the right parameters, the software runs the function and returns the result to the model, which continues its reasoning. It is the mechanism that turns a chat into an agent: gpt-oss, Qwen and Gemma support it, and engines like vLLM and Ollama handle it natively.

Neural network architecture introduced in 2017 (Attention Is All You Need) that revolutionized NLP. Unlike recurrent networks, Transformers use attention to process the entire input simultaneously. All modern LLMs (GPT, Llama, Mistral, Claude) are based on this architecture.

Apple's technology that fuses two M5 Max dies into a single M5 Ultra chip, presented to the system as one processor. The unified memory of both dies forms a single pool (up to 512GB) accessible to CPU, GPU and Neural Accelerators.

The Apple Silicon architecture in which CPU and GPU share the same pool of RAM. For local AI, the entire model can reside in that memory without copying data between system and video memory.

A specialized database that stores document embeddings (numeric vectors) and answers similarity searches: given the embedding of a question, it returns the closest passages. It is the component that makes RAG possible.

Voiced Language Model with a Varying-rate adversarial (VITS): a neural text-to-speech architecture that generates the waveform directly from text without a separate vocoder. It underpins Piper and Kokoro-82M.

Voice Activity Detection: a technique that identifies which portions of an audio stream contain speech, so the audio can be segmented into relevant chunks and silence ignored. Essential in real-time transcription pipelines.

Video RAM. The dedicated memory of the graphics card (GPU). It is the critical resource for local inference: a 7B model quantized at 4-bit requires about 5 GB of VRAM, a 13B model about 9 GB. When VRAM is insufficient, the system can use system RAM, but with reduced performance.

Khronos's cross-platform GPU programming API, born in gaming but now the open path for local AI: Ollama and llama.cpp support it natively on Windows, Linux and macOS (via MoltenVK), on NVIDIA, AMD and Intel with no proprietary drivers. Less optimized for inference than CUDA — usually 70-90% of the native path's speed — but it is the only stack that works on practically every card, and it is the maintained path for Intel Arc GPUs since IPEX-LLM was archived.

Word Error Rate: the standard metric for comparing transcriptions. It compares the produced text against a reference transcript: fewer errors per word means a lower score and a better model.

A sequence of repeatable steps that connects systems together: an event arrives, operations run in order, a result is produced. In automation platforms, the workflow is built visually by connecting nodes (email → AI model → CRM → notification).