Skip to main content
← Back to news

Local model won't stop talking? The fault is often the chat template

Tools & releases
Source:NobodyWho
A well-argued article explains why a local model that stalls, ignores the System prompt or keeps generating past its answer is almost always a chat-template or Token-metadata problem in the GGUF file, not a weights problem — and how to fix it with GGUF-surgeon without re-quantizing.
Read original article

Related articles

GGUF: The Format That Keeps Models on Your Hardware

We open the box: what's really inside a .gguf file, why it's a single file, how to read the filename, and how to pick the size for your VRAM, with the real sizes of an 8B model.

Decision models in local: JEV-35B and the 3B-active System 1

You don't need a big LLM to decide. JEV-35B (35B, 3B active) in NVFP4 on a 32GB GPU and ej (11.4MB) on CPU show how to do routing, classification and quality control without generating a single token.

Costs: local vs API, done with real numbers

How much does local really cost compared to a cloud API? Verified per-token prices, GPU amortization, electricity and the break-even point, calculated with real numbers.

Get the next local-AI briefing

Once a week, the top selection from iaLocale: articles, news and resources on local AI, and more.

After signing up you receive a confirmation email (double opt-in). Without confirmation we send nothing. No spam, we promise.

Related resources

Tools and resources from the iaLocale archive relevant to this content.

Bespoke Nimble 9B

Models

Bespoke Labs' 9B decision model: it runs entirely locally via Ollama 0.35 and answers typed questions (true/false, numeric scale, multiple choice) through the new systemone API. Built for fast classification in automated pipelines, no cloud.

⚡

faster-whisper

Inference

Reimplementation of Whisper based on CTranslate2: up to 4 times faster than openai/whisper with the same accuracy and less memory. Supports 8-bit quantization on both CPU and GPU.

🎵

Kokoro-82M

Models

82-million-parameter speech synthesis model on a VITS architecture, Apache-2.0 for the weights, with voices in dozens of languages and surprisingly high quality for its size. Runs efficiently on CPU alone.

💻

LM Studio

Software

Terminal-free desktop app for downloading and running GGUF models: GUI, built-in model search and a local server. The simplest way to get started.

Browse all resources