Local AI on Raspberry Pi

A Raspberry Pi is not a server, but it can run local AI models — and for a precise subset of jobs it does the work better than you would think: always-on personal assistant, light automations, classification, prototyping. The point is not speed, it is price: a machine running 24/7 costs less than a coffee a month in electricity. Here are the real numbers and the path from first install to a model that answers.

What you can really do with it, and what you cannot

The physical rule is simple: generation speed is limited by memory bandwidth, not by CPU cycles. The formula:

max tok/s ≈ memory bandwidth (GB/s) ÷ model file size (GB)

The Pi 5 has about 17 GB/s of bandwidth. A Llama 3.1 8B at Q4_K_M weighs ~4.9 GB: the theoretical ceiling is 17 ÷ 4.9 ≈ 3.5 tok/s; measured on real Pi 5 boards it settles between 2 and 3 tok/s (Jeff Geerling recorded 1.99 tok/s on a 16GB). In practice:

Model (Q4)	File	Pi 5 theoretical ceiling	Pi 5 measured	Use
Llama 3.2 1B	~0.8 GB	~20 tok/s	10–15	automations, classification
Llama 3.2 3B	~1.9 GB	~8.6 tok/s	4–6	light chat, text reorganisation
Qwen3-4B	~2.5 GB	~6.8 tok/s	3–5	chat with better Italian and French
Llama 3.1 8B	~4.9 GB	~3.5 tok/s	2–3	maximum usable quality
Qwen3-8B	~5.3 GB	~3.2 tok/s	2–3	same, multilingual

3 tok/s means one sentence every 3–4 seconds: fine for interactive chat, not for analysing a 50-page document. MoE models (Qwen3-30B-A3B) do not fit by size: 19 GB at Q4, and the 16GB Pi 5 does not carry it.

Which Pi to buy

Prices changed a lot in 2026: memory cost doubled on AI infrastructure demand, and Raspberry Pi raised prices twice in three months.

Board	Official price (2026)	Note
Pi 4 (8GB)	~$115	25 GB/s bandwidth, slower CPU: 1.5–2 tok/s on 8B
Pi 5 (8GB)	~$125	Works for 1–4B; the 8B Q4 (4.9 GB + KV cache + OS) fits, but barely
Pi 5 (16GB)	~$205	The right pick: 8B Q4 with margin, 8–16K context

Official prices are published by Raspberry Pi in its price-rise notes; the values above reflect the two increases (December 2025 and February 2026). Add to the kit: the official 27W power supply (the Pi 5 wants 12–15W under full load), a fan case, and an NVMe SSD if you want decent I/O ($40–60 of accessories).

From first install to a model that answers

1. Prepare the board. Download Raspberry Pi Imager, select Raspberry Pi OS (64-bit), and write the image to the microSD. During configuration, enable SSH and set a name. If you write to NVMe over USB, boot is faster and swap does not wear the microSD.

2. Power on and verify. Connect HDMI, keyboard, network. At first login:

sudo apt update && sudo apt full-upgrade -y
free -h

You should see free RAM (on a 16GB, ~15 GB available). If the Pi reports "undervoltage", you have the wrong power supply: move to the official 27W.

3. Install Ollama.

curl -fsSL https://ollama.com/install.sh | sh

When done, ollama --version should print the version. On a Pi 5 the install is clean: ARM64 is supported directly.

4. Pull the model.

ollama pull qwen3:4b
ollama run qwen3:4b

First run downloads ~2.5 GB: at 50 Mbps that is about 10 minutes. When the prompt appears, try: "Write in English: what is the difference between RAM and VRAM, in two lines."

5. Verify it runs where it should.

ollama ps

It should show the model at 100% CPU. If the model disappears and reappears, it is going to swap: the file does not fit in RAM and speed collapses to 0.5 tok/s. If the model is "100% CPU" and speed is the expected one (3–5 tok/s for a 4B Q4), you are fine.

What it costs in electricity

The Pi 5 under full CPU load consumes 12–15 W; at idle, 3–4 W. The math:

Scenario	Average draw	kWh/month	Cost (€0.25/kWh)
24/7, mixed load (avg 8W)	8 W	5.8	~€1.45/month
24/7, with NVMe and peripherals (avg 15W)	15 W	10.8	~€2.70/month
12 h/day (avg 8W)	8 W	2.9	~€0.72/month

Calculation: 8 W × 24 h × 30 d = 5.76 kWh; × €0.25/kWh = €1.44. That is an order of magnitude below any desktop: a workstation with an RTX 4090 under load draws 350–450 W and costs ~€40/month at 8 hours of work, as shown in What a token costs.

The three tricks that matter

1. NVMe for swap. The Pi 5 supports NVMe over USB: put swap there and the microSD does not wear. The loaded model stays in RAM; the NVMe serves boot and swap of secondary processes.

2. zram to extend RAM. If the model sits at 5.5 GB on an 8GB board, zram compresses free RAM and gives you margin:

sudo modprobe zram num_devices=1
sudo zramctl /dev/zram0 --size 4G
sudo mkswap /dev/zram0 && sudo swapon /dev/zram0

3. Control the context. Every context token adds bytes to the KV cache. On a 4B, 8K of context cost ~1 GB; 32K costs 3–4. For short chats, keep the default (4–8K): you do not need more, and every GB the context takes is a GB less for the model.

Where the Pi is not the right choice

If your case is quality (long reasoning, complex code, articulated writing), the Pi is not the machine: an 8B at 3 tok/s is not an experience. If your case is speed (document analysis, batch automations), 10–20 tok/s on a desktop CPU at €150–200 is 5–8 times faster at a quarter of the price. The Pi wins in one precise case: always on, always available, negligible electricity cost, zero footprint.

In short: the 16GB Pi 5 (~$205) runs Qwen3-4B at 3–5 tok/s and Llama 3.1 8B at 2–3 tok/s, at an electricity cost of €1–3 per month at 24/7. It is an always-on personal assistant, not a workstation. For quality or speed, buy another machine; for "always on", the Pi is unbeatable.

Sources

• Raspberry Pi — official prices and price-rise notes (December 2025, February 2026)
• Tom's Hardware — Pi 5 16GB at $205, second increase in three months
• Ollama — Linux install
• Raspberry Pi Imager — download
• Pi 5 benchmarks: theoretical ceilings and measured values per model
• Jeff Geerling — Llama 3.1 8B on a 16GB Pi 5: 1.99 tok/s

Article generated with AI