Blog
Articles, guides, and insights on local artificial intelligence
Reading paths
Start here
The in-order path for newcomers to local AI: from basic concepts to installation and your first model.
Optimize performance
Pick the right hardware and get the most out of what you already own: CPU vs GPU, Mac, Raspberry Pi, and fast serving.
Data and privacy
What actually goes over the network with local AI: the data, the security, the compliance, and the control that stays in your hands.
Choosing hardware and model
How to pick the right hardware and the model suited to your use: VRAM, quantization, GPU or CPU and response speed.
36 articles
ML agents that run locally: MLExpert and EpochGo on Ollama, zero API costs
MLExpert raises the bar to 35B and a 72.7% medal rate; EpochGo runs three AutoML agents on 16 GB at zero API cost: what works today, the math and the limits.
Intel and AMD collaborate on PyTorch/oneDNN: what it means for local AI on CPUs
AMD is merging ZenDNN into oneDNN and Intel is folding in its ACE extensions: one open-source x86 acceleration stack in PyTorch, already the default. What it changes if you run models locally on CPUs.
MiniCPM-V 4.7: the 35B MoE that runs on a smartphone with 3 GB of RAM
MiniCPM-V 4.7 (35B total, ~3B active, released October 12, 2026 by OpenBMB and ModelBest) is the first open MoE in the series, vision-language: multiple images, video, OCR. 13+ tok/s with 3 GB of RAM on an iPhone 17 Pro Max, 17+ on Android. Official quantizations BF16, FP8, INT4, AWQ, GPTQ, GGUF; engines Ollama, llama.cpp, vLLM, SGLang. The three techniques: Lookahead Expert Scheduling, fine per-level quantization, DSpark speculative sampling.
Ling 3.1 Flash: 560B open-weight with MIT license and 1M context — what changes if you have 24GB of VRAM
inclusionAI's Ling 3.1 Flash: hybrid MoE 560B total / 25B active, native 1M-token context, FP8 checkpoint, MIT license. What it concretely means for 24GB of VRAM: quantizations, parallelism and API costs.
Aleph Alpha's Kolibri: the 78B MoE that runs (almost) locally
Aleph Alpha's Kolibri (78.1B/3.46B, Apache 2.0, FP8) is the European open-weight MoE: reasoning, tool calling and context up to 1M tokens. In FP8 it weighs ~78GB of VRAM: it runs on an RTX 5090 32GB + shared RAM or on unified-memory Macs; for practical adoption the kolibri1 architecture in llama.cpp and Ollama/LM Studio builds are still missing.
Decision models in local: JEV-35B and the 3B-active System 1
You don't need a big LLM to decide. JEV-35B (35B, 3B active) in NVFP4 on a 32GB GPU and ej (11.4MB) on CPU show how to do routing, classification and quality control without generating a single token.
Mac Studio M5 Ultra: 512GB of unified memory on the desktop
Four dies, 512GB of unified memory at 1.2 TB/s: the Mac Studio M5 Ultra runs 100-560B MoE models locally. The speed math, the cost comparison with the DGX Station, and when the Mac is the right choice.
What a token costs: five routes, the math, and the break-even point
Frontier API, subscription, inference providers, rented GPU or your own hardware: five routes to tokens, with real monthly costs, the electricity bill, and the calculated break-even point.
NVIDIA, AMD and Intel for local AI: which platform to choose
CUDA, ROCm or Vulkan? We compare the three local-AI platforms based on what is actually available: official software stacks, cards by VRAM tier (12, 16 and 24GB) and real speeds. Where the obvious choice remains, where AMD wins on value, and when it makes sense to leave NVIDIA.
The 16GB Blackwell cards: 5060 Ti, 5070, 5070 Ti
The 16GB VRAM tier is the center of gravity of local AI in 2026. It's enough memory to run the best open dense models in Q4 formats — the 27B class, the new balance point of the open catalog — without giving up too much context, and a list price that (depending on the month) stays within a personal budget. The Blackwell generation offers three cards exactly in the middle of this tier: the RTX 5060 Ti, 5070 and 5070 Ti. Same generation, same GDDR7, three very different experiences. Here's how they compare: what each one really brings you, and where your budget should go.
Speculative decoding and MTP: speed for free
Speculative decoding makes local AI faster without changing output quality: a small model drafts the tokens, the big model verifies them. Here's how the draft-verify loop works, MTP heads in Qwen models, EAGLE on Mistral Medium 3.5, and how to enable it all in three commands.
gpt-oss: OpenAI's open reasoning model
21B total parameters but only 3.6B active, native MXFP4 weights and 128K of context: OpenAI's first open reasoning model fits in 16GB. What that means in practice.
Unified memory: from Strix Halo to Mac Studio M5 Ultra — what fits and at what speed
128GB or 512GB of memory shared between CPU and GPU: three real platforms, the bandwidth that determines speed, and the math on what fits and for whom it makes sense.
Local Image Models: from SDXL to Qwen-Image
From SDXL's 2.6B parameters to Qwen-Image 2.1's 7B: how diffusion models work, how much VRAM you actually need, and which model to pick for your hardware.
Qwen3.8-27B vs Gemma 4 26B A4B: the new frontier at 24GB
A 27.8B dense or a MoE with 3.8B active, a 262K context against 256K and almost the same weight at Q4_K_M: two opposite ways of putting big-model quality on a 24GB consumer card. Which is faster, which goes deeper, and when to pick one over the other.
Ollama vs llama.cpp: under the hood
Ollama runs llama.cpp under the hood, but the two are not the same thing: one is the engine, the other the car. What's really underneath, when the difference matters, and how to choose without slogans.
Costs: local vs API, done with real numbers
How much does local really cost compared to a cloud API? Verified per-token prices, GPU amortization, electricity and the break-even point, calculated with real numbers.
Local voice: from Whisper to text-to-speech
How to build a fully local voice pipeline: transcription with Whisper, synthesis with Piper or Kokoro-82M, hardware requirements, and practical use cases.
Italian in local models: what to expect
How many local models actually handle Italian well? The vendor numbers, where they fail, and the half-hour method to measure them yourself.
Privacy: what actually goes over the network when AI is local
Your data doesn't leave the machine — almost. Here's what actually goes out over the network with Ollama and LM Studio, and how to verify it in half an hour with tcpdump, Wireshark and mitmproxy.
A Model Without a GPU: When the CPU Is Actually Enough
In 2026, CPU-only inference is no longer a fallback: when it genuinely works, what speed to expect, and which levers to pull to squeeze the most out of your processor.
KV cache: why your RAM needs more than just the model
What the KV cache is, the formula to calculate it with real numbers from Llama 3.1 8B and Qwen3 8B, and the three levers to manage it: context, quantization and flash attention.
GGUF: The Format That Keeps Models on Your Hardware
GGUF is the single-file format that ships models for local inference: a header with a signature and version, key-value metadata, a tensor index, and weights aligned for fast mmap loading. Quantization types with real sizes (8B from 14.96 GiB at F16 to 4.58 GiB at Q4_K_M), the filename convention, and the rule for picking a size against your VRAM or RAM.
Fine-tuning: adapting a model to your data with LoRA and QLoRA
Fine-tuning with LoRA and QLoRA: when it's truly needed, how to prepare 100-500 quality examples, how much VRAM it takes (8-12 GB in QLoRA for a 7B), and which local tools to use: LLaMA-Factory, Unsloth, MLX.
Quantization: Get More From the Same Hardware
Quantization reduces the precision of the weights (from FP16 to 4–8 bit) so models fit in less VRAM and generate more tokens per second. The levels, the real file sizes of Qwen2.5 14B, the rule for choosing from Q4_K_M, and the limits (code, reasoning, languages other than English).
Qwen-Image 2.1: Generate and Edit Images Locally
Qwen-Image 2.1 unifies text-to-image, editing, and RGBA transparency in a 7B open model (32-layer DiT). Local setup with diffusers, native ComfyUI support, VRAM needs (~14GB in bfloat16), and use cases (stickers, product photos, design workflows).
When a Big Model Isn't Needed: Choosing by Size
The method for choosing model size by hardware: the three limits (memory, speed, perceived latency), the VRAM table for Q4 quantization, the hardware classes, and the cases where the small model wins.
How to Read a Model's Spec Sheet: Family, Size, Context
A guide to reading a model's specs: family, total and active size, context window, quantization, benchmarks and license, with a worked example on DeepSeek V4.1 Flash.
GPU or CPU: when the CPU is enough
How prefill and decode work, what matters for speed (memory bandwidth), real numbers on a mainstream laptop, the cases where the CPU is enough and where VRAM is needed — with the commands to measure.
Open WebUI: the interface for your local AI
Open WebUI is the full web interface for your local models: Docker install, connecting Ollama, per-model customization, document RAG, and its honest limits.
Local AI on Mac: the complete Apple Silicon guide
Why Apple Silicon (unified memory, GPU cores) is the simplest machine for local AI: the RAM you need, installing Ollama, recommended models per chip generation, and realistic performance expectations.
Hardware for Local AI: What You Need in 2026
The memory rule (weights + KV cache), real October 2026 prices for GPUs, Apple Silicon and CPU, the 36-month cost math and the choice by budget.
Running LLM Models Locally: From First Install to First Working Model
Learn how to install and use large language models on your own computer, without relying on cloud services.
Local AI on Raspberry Pi
From first install to a model that answers on a Pi 5: real 2026 prices, the tok/s = bandwidth ÷ file formula, the electricity bill and the tricks that matter.
Ollama vs LM Studio: Which Tool to Choose
Ollama for servers and scripts, LM Studio for desktop exploration: the comparison table, the two APIs (11434/1234) and the real case of using both.
How to Choose Your First Local AI Model
The three criteria for picking your first local model: use case, memory (with the exact formula) and context, with the recommended models for 8, 16 and 24 GB.