llama.cpp

The C/C++ engine behind much of local AI: runs quantized models on almost any hardware, with a command-line tool and an OpenAI-compatible server.

  • Run locally
  • MIT · Open source
  • macOS, Windows, Linux
  • Verified

llama.cpp is an open-source inference engine written in plain C/C++ with no dependencies. Its goal, in its own words, is to run language and vision-language models with minimal setup and strong performance on a wide range of hardware, locally and in the cloud. Apple silicon is a first-class target, and it also supports x86 CPUs with AVX and AMX, NVIDIA GPUs through CUDA, AMD GPUs through HIP, and Vulkan, SYCL, Metal, OpenCL and other backends.

Its main trick is quantization: weights can be stored with 1.5 to 8 bits per value, which shrinks memory use and speeds up generation, and CPU+GPU hybrid inference lets you partly accelerate a model that is larger than your video memory. Models are distributed in the GGUF format. The llama cli command downloads and runs a model straight from Hugging Face with -hf, and llama serve starts an OpenAI-compatible API server with a built-in web interface.

Many friendlier tools build on it. LM Studio says it uses llama.cpp and MLX under the hood, and Unsloth uses it to run and export models. If you want the most control, or the lowest overhead, or you need to run a model on unusual hardware, use llama.cpp directly.

Good for

  • Running models on CPUs, older GPUs, Apple silicon and other unusual hardware
  • Squeezing a large model into limited memory with 2-bit to 6-bit quantization
  • Building a small, dependency-free local server with an OpenAI-compatible API
  • Learning how local inference actually works

Watch out for

  • It is a command-line tool, so it has a steeper learning curve than Ollama or LM Studio
  • Heavier quantization saves memory but lowers quality, so test your task at the precision you plan to use
  • Support for a brand-new model architecture can lag behind its release

Links