Tools, frameworks, and models for local AI
52 resources found for "LLM"
Alibaba Group: the Qwen models are among the world's most performant open-weight LLMs, released under permissive licenses. The Qwen models in the catalog are the child resources.
27B reasoning model with ternary weights at 1.72 bits per weight, derived from Qwen3.8-27B with hybrid attention (~75% linear). In GGUF PTQ1_0 it takes 5.95 GB, retains 98.2% of the FP16 intelligence and reaches ~47 tok/s on an Apple M5 Max laptop, with a 262K-token context. Apache 2.0 license. Requires Prism ML's llama.cpp fork (ternary kernels are not in stock llama.cpp).
DeepSeek (Shenzhen, China): the most efficient open-weight LLMs in their class, with a focus on low-activation MoE. DeepSeek V4.1-Flash pushes 552B total parameters with ~8B active and a 1M context. The models in the catalog are the child resources.
DeepSeek's MoE model with 552B total parameters and 8-16B active per token. 1M-token context, MIT license: free commercial use. Built for online serving, too large for local use on a single PC.
Google's open model family in 1B, 4B, 12B and 27B sizes. 128K-token context, 140+ supported languages and multimedia input (text and images). An excellent choice for RAG and long documents on consumer hardware.
La taglia compatta della famiglia Gemma 3 di Google: 4,32B di parametri, contesto di 128K token e supporto nativo a 140+ lingue (incluso l'italiano), con input testo+immagini. Licenza Gemma (uso commerciale con condizioni). In Q4_K_M occupa circa 2,9GB: gira su GPU da 8GB e su CPU moderne.
La taglia 12B della famiglia Gemma 4 di Google: 11,95B dense, contesto 256K e multimodalità integrata senza encoder separati (testo+immagini+audio). Licenza Apache 2.0. In Q4_K_M occupa circa 7,4GB: il punto di equilibrio tra qualità e requisiti hardware per l'uso locale su 8-16GB.
Il modello MoE della famiglia Gemma 4: 25,2B di parametri totali con 3,8B attivi per token. Contesto 256K, input testo+immagini, oltre 140 lingue, Apache 2.0. In Q4_K_M pesa circa 16,8GB: la scelta per chi ha 24GB di VRAM e vuole qualità da modello grande con velocità da modello piccolo.
Compact model from Google's Gemma 4 family: 8B total parameters, of which 4.5B effective thanks to per-layer embeddings (PLE). 131K-token context, 140+ languages, text+image and audio input. The entry size of the Gemma 4 family for local use on 8-16GB.
Modello MoE open-weight di inclusionAI: 560B parametri totali, 25B attivi per token, architettura ibrida (attenzione lineare KDA + Gated MLA), finestra di contesto fino a 1M di token, licenza MIT. Reasoning ibrido con modalità di pensiero on/off. A 4 bit entra in 512GB di memoria unificata.
Intel: Arc (Xe) GPUs are the third consumer platform for local AI, with 12-16GB of VRAM at entry prices. The maintained inference path is Vulkan (native Ollama, llama.cpp): the IPEX-LLM project was archived in January 2026. The GPUs in the catalog are the child resources.
La prima scheda Intel da prendere sul serio per l'IA locale: Xe2 Battlemage, 12GB GDDR6 a 456 GB/s, 190W. 12GB a prezzo d'ingresso: i modelli 8-14B in Q4 stanno interamente in VRAM. Il software è il punto debole: IPEX-LLM è stato archiviato a gennaio 2026, il percorso mantenuto oggi è Vulkan (Ollama nativo, llama.cpp) con ~40-42 tok/s su 8B Q4.
Framework for building LLM-based applications with chains, agents and tool use.
Meta's largest open model: closed-frontier quality but datacenter requirements. In practice it needs high-VRAM multi-GPU or the cloud.
Frontier-class open model with 131K-token context. Needs 48-80GB VRAM, a Mac with unified memory, or a multi-GPU setup.
The de-facto local workhorse: solid quality, 131K-token context and a huge GGUF quantization ecosystem. The standard choice for chat and assistants.
Small, fast model with up to 131K-token context. Ideal entry point for chat and summarization on modest hardware, great on phones and mini-PCs.
Il modello MoE grande della famiglia Llama 4 di Meta: 400B di parametri totali con 17B attivi (128 esperti). Contesto 1M di token, nativamente multimodale (testo+immagini), licenza comunitaria Meta. Richiede multi-GPU o macchine con memoria unificata 128GB+ anche quantizzato.
The compact MoE of the Llama 4 family: 17B active parameters out of 109B total (16 experts). Meta's community license allows free use under 700 million monthly users. One of the best entry points into the Llama 4 family.
Open source framework for unified fine-tuning of 100+ LLMs and VLMs (Qwen, Llama, Mistral, Gemma): from full SFT to LoRA, QLoRA and DPO, with a built-in WebUI and startup from a single YAML file. The community's reference project (ACL 2024).
C++ implementation of LLMs for efficient inference on consumer CPUs and GPUs.
Open-source tool that estimates local model inference speed across 250+ hardware configurations (GPU, Mac, CPU): measured estimates from community benchmarks or derived from similar setups. Use it to choose hardware before buying.
Open-source inference engine (Rust, YC S25) that compiles and tunes its kernels directly on the target hardware: Apple Silicon, NVIDIA, AMD or CPU. On open models it claims up to 2× the speed of llama.cpp. Also aimed at serving agents.
Meta Platforms: the Llama family is the world's most widely used open-weight LLM line, from 1B to 405B parameters, with a commercial-use license. The Llama models in the catalog are the child resources.
The first open-source MoE of OpenBMB's MiniCPM-V series: 35.2B total parameters, ~3B active per token, multi-image and video understanding with early visual-token compression (LLaVA-UHD v4). First open-source MoE with fluid on-device decoding — 13+ tok/s on 3GB of RAM on an iPhone 17 Pro Max — thanks to Lookahead Expert Scheduling (llama.cpp-omni), fine per-level quantization and DSpark speculative sampling. Day-0 support for Rockchip, Houmo, Intel, AMD and Arm; runs on Ollama, llama.cpp, vLLM and SGLang.
Fast, efficient 7B model, well-tuned for multilingual European tasks. Excellent instruction quality at a fraction of the cost of 13B+ models.
Mistral AI (Paris, France): the most efficient European open-weight LLMs, with permissive licenses and a focus on privacy and localization. The Mistral models in the catalog are the child resources.
Il primo 'modello unificato' di Mistral: dense da 128B che unisce instruction-following, reasoning e coding in un'unica serie di pesi, sostituendo Medium 3.1, Magistral e Devstral 2. Multimodale (testo+immagini), contesto 256K, ragionamento configurabile. Open weights con licenza Modified MIT: uso libero fino a una soglia di fatturato per le grandi aziende.
Il modello della linea Small di Mistral, ora MoE: 119B di parametri totali con 6,5B attivi per token (128 esperti, 4 attivi). Unifica instruct, reasoning e coding in un unico modello multimodale (testo+immagini) con contesto 256K. Licenza Apache 2.0.
Eight 7B experts with two active per token: 47B total params with large-model reasoning at a fraction of the compute cost. An open-world classic.
Apple's machine learning framework for Apple Silicon: lazy arrays computed across unified CPU and GPU, with unified memory as an asset. The mlx-lm library brings LLM training, fine-tuning (LoRA/QLoRA) and inference directly to the Mac.
The runtimes that run local models on your hardware: Ollama for simplicity, llama.cpp for portability, vLLM for production throughput. The engines in the catalog are the child resources.
NVIDIA's vision encoder-decoder family for local document understanding: markdown, LaTeX formulas, bounding boxes and a semantic class per element, in reading order. ViT-H (C-RADIO) + 10-block mBART decoder, all under 1 billion parameters. Locally via vLLM (v0.20–v0.26 for 2.0) or Transformers, on NVIDIA GPUs (Turing through Blackwell).
11B end-to-end full-duplex voice model: audio understanding and generation in a single architecture, without the cascaded ASR→LLM→TTS stack. First open full-duplex model with tool calling during conversation, ~450ms responses, interruptions, and natural turns.
Il modello di ragionamento open-weight di OpenAI: MoE con 21B di parametri totali e 3,6B attivi (32 esperti, top-4), contesto 128K e quantizzazione MXFP4 nativa. Licenza Apache 2.0: i pesi partono già quantizzati a 4 bit, senza passo di conversione. Sta in 16GB di VRAM e gira anche su RAM unificata o CPU.
Cloud gateway for LLM inference: a single API endpoint for hundreds of models (including open-weight MoE), with per-token pricing and automatic routing across providers. For Ling 3.1 Flash: $0.075/M input, $0.225/M output.
7B open-weight model (32-layer DiT) that unifies text-to-image generation up to 2048x2048, editing with up to 10 reference images, and native RGBA transparency. Designed to run locally on 16GB+ VRAM; supported by diffusers and ComfyUI.
Ultra-compact 0.5B-parameter model. Runs on any hardware, even embedded: ideal for lightweight summarization and low-power local devices.
Clear step up in reasoning and instruction quality over 8B models. Excellent for coding, agents and structured writing on 16GB GPUs or Macs with 32GB RAM.
Near-70B quality at a fraction of the size: the sweet spot for advanced use. Excellent in Italian, code and reasoning. Needs 24GB+ VRAM.
3B-parameter model with excellent multilingual support, including Italian. Good balance of quality and hardware requirements for summarization and drafting.
The top open dense model of its class, with first-class multilingual support for Italian, French and other European languages. Needs 48-80GB setups.
Mixture-of-Experts model: 30B total params but only 3.3B active per token. Small-model speed with mid-size quality; the best speed/quality ratio in its class.
Il modello dense 8B della famiglia Qwen3: 8,19B di parametri, contesto nativo di 40K token (espandibile con YaRN) e ragionamento in modalità think/non-think attivabile per prompt. Licenza Apache 2.0. È il punto di equilibrio per l'uso locale su GPU da 8-12GB: in Q4_K_M occupa circa 5GB, lasciando spazio per contesto lungo.
A 35B-parameter MoE model with ~3B active per token from the Qwen family, Apache-2.0. It is the fine-tune base of JEV-35B. Suited to 40GB+ GPUs in BF16 or to 4/8-bit quantizations for smaller hardware.
Il modello dense multimodale della generazione Qwen3.8 di Alibaba: 27,78B di parametri, input testo+immagini+video e contesto nativo di 262K token. Licenza Apache 2.0. L'attenzione ibrida (DeltaNet lineare + GQA) riduce la KV cache di circa il 75% rispetto a un transformer standard: in Q4_K_M sta su una singola GPU da 24GB.
Open-source serving engine for LLMs with RadixAttention, continuous batching and first-class speculative decoding (MTP, EAGLE, draft models, ngram). Apache-2.0 license.
A zero-dependency Python tool that recognizes and summarizes every file with a local LLM (Ollama, LM Studio or MLX-LM), fully offline. Detects type by content, extracts structure (CSV, email, scripts, logs) and reports files carrying instructions to the assistant. Works without a model (--no-llm). MIT license.
100% local macOS dictation app: on-device transcription with Whisper and a rewrite pass with a local LLM (Qwen) on the M-series GPU. Text lands in the target app; no data touches the cloud.
Optimized fine-tuning framework for NVIDIA GPUs and Apple Silicon: custom CUDA matrices that cut memory usage and double training speed versus standard methods. Supports LLMs and diffusion models.
High-throughput LLM serving with PagedAttention and hardware optimizations.
