News

Moose 0.6 adds protected credential storage for cloud services, a Local-Only mode with an app-managed Ollama (model download and removal), per-chat or per-provider permissions, a GNOME Files browser to attach documents to conversations, and an expanded local model catalog. 0.6.1 adds Mistral and DeepSeek. Available on Flathub.

The KyutaiSTT module brings Kyutai's streaming speech recognition to Apple Silicon via MLX. The reference checkpoint is the STT-1B (English and French) quantized to 8-bit, with a causal Mimi encoder producing 32 audio codes every 80ms. CC-BY-4.0 license.

Google DeepMind and GovTech Singapore use lightweight on-device Gemma models to intercept scams in private chats, without sending data to the cloud.

Open-source project on GitHub: a local tutor that studies the books and documents you give it and teaches the content with quizzes and questions, using a local LLM. Designed for people who want to learn without sending their materials to a cloud.

The Google AI Edge team released ML Drift, a cross-platform GPU compute engine for on-device Inference under the Apache 2.0 license. It covers OpenGL ES, OpenCL, Metal and WebGPU, supports 5D tensors for 3D networks and generative models, and is the new acceleration core of LiteRT.

A native macOS app turns dictation into text with Whisper running on-device plus a local LLM (Qwen) rewrite pass on the M-series GPU: no byte leaves the computer, no cloud account. Audio goes through Whisper, then the local model cleans up punctuation and formatting before delivering text to the target app.

On GitHub, an experimental project offloads part of local LLM Inference to a paired iPhone, using the phone's GPU and memory as a secondary accelerator — an idea for anyone wanting more tokens per second without swapping graphics cards.

llama.cpp b11337 widens the CPU/edge backend matrix: Snapdragon (Adreno and Hexagon), OpenVINO, SYCL FP16/FP32, CUDA 13, Android, and ROCm 10, plus server fixes. It is the eighth release in the October 2 batch.

The vLLM project tags proto-v0.4.0, the experimental development track that precedes the stable releases of the language-model Inference engine. One more step for anyone serving local models in production.

A well-documented experiment: 67 course videos transcribed locally and offline with whisper-large-v3-turbo rebuilt on MLX for the Mac's GPU — 17 hours of video processed in 34 minutes, without a single byte leaving the machine.

Ollama's v0.34.4 release applies structured outputs on thinking models in a single pass (faster and more reliable), fixes 'model not found' errors with large local libraries, and speeds up Qwen 3.8 prompt processing on Apple Silicon. A useful update if you run Ollama in production.

The weekly report on the llama.cpp project covers the current build b4991 and the week's activity: a bug where Gemma 4 in Thinking mode emitted progressively longer trailing garbage was resolved by trimming whitespace in the chat template; the Vulkan CI runners moved to the NVIDIA r615 driver, and the report analyzes the community's most active issues.

The vLLM project released version 0.29.0, with 594 commits from 277 contributors. The main highlight is Model Runner V2, now the default for all models, with CUDA graph memory profiling for KV cache auto-sizing. New model support also lands: Qwen3.8-Flash-Next, Hy4-preview (a 770B MoE with native MTP), Granite SWA, and Kimi K3 checkpoints in NVFP4.

Gemma4 can now process images and audio on the MLX engine. Also, cached prompt tokens are now reported correctly and default parameters defined in GGUF files are honored.

New Qwen3.8 Flash Next support on the MLX engine, structured output for mlxrunner, and a fix for Metal GPU timeouts when loading models from slow storage.

Apple updated the MLX framework with native support for 4-bit quantized GGUF models, allowing larger models to run on M-series chips.