llama.cpp

Library written in C++ to run LLM models on consumer CPUs and GPUs. It is the underlying engine for many local inference tools, including Ollama. It supports GGUF quantization and can run large models on limited hardware by leveraging both CPU and GPU.