Local AI Glossary

The splitting of a document into small text blocks before indexing. Each block becomes a searchable entry in the vector database: block size is one of the first parameters to tune when RAG results are not convincing.

Family of speculative decoding techniques (EAGLE paper, arXiv:2401.15077): a small draft model reads the big model's hidden states and predicts the next tokens from there. Sampling stays lossless, like classical speculative sampling. Official EAGLE checkpoints (for example Mistral-Medium-3.5-128B-EAGLE) are served natively by vLLM and SGLang.

The art of writing effective instructions to get better results from a language model. Includes techniques like few-shot prompting (providing examples in the prompt), chain-of-thought (asking the model to reason step by step) and role prompting (defining a role for the model).

Training that simulates quantization during training, so the model learns to work with low-precision weights. Unlike post-training quantization (applied cold, e.g. Q4_K_M), QAT preserves much more quality at the same bit width. Google released official QAT checkpoints for the Gemma 4 family in June 2026.

The closeness between two texts' vectors based on meaning, not on literal match. It is the principle behind search in a vector database: "brake" and "braking system" are not the same word, but their vectors are neighbors.

The set of instructions that defines a model's permanent behavior: tone, role, response format, constraints. Unlike the user prompt (a single question), the system prompt governs every response of the session and is not shown to the model as a request.

Voice Activity Detection: a technique that identifies which portions of an audio stream contain speech, so the audio can be segmented into relevant chunks and silence ignored. Essential in real-time transcription pipelines.