On-device AI

On-device AI means the model runs on your own phone, laptop or small board instead of in the cloud, so data stays local and it works offline.

  • Using models
  • Updated

Running locally avoids sending private text, audio or images to a server, works without a connection and has no per-request fee. The limits are memory and speed: phones and laptops have a few to a few dozen gigabytes, so on-device models are small, roughly 1B to 10B parameters or mixture-of-experts models that activate only a few billion, and are usually quantized to 4-bit.

Examples on this site include the E2B and E4B models of Gemma 4, Liquid AI's LFM3, the 1B model of MiniCPM-V 4.7 and Mistral's Voxtral Realtime for speech. Runtimes such as llama.cpp, MLX and Ollama handle the packaging. Battery use and heat matter on phones, so measure speed on the actual device.

Related terms

  • Quantization Quantization stores a model's weights with fewer bits, for example 4 instead of 16, to cut memory use and speed up inference at a small cost in quality.
  • Unified memory Unified memory is a design, used by Apple silicon and some other chips, in which the CPU and GPU share one pool of memory, so a large model can use most of it.
  • GGUF GGUF is a single-file model format used by llama.cpp and the tools built on it, which bundles quantized weights with the information needed to run them.
  • VRAM VRAM is the memory on a graphics card; it decides which models a GPU can hold, because the weights must fit in memory for the model to run quickly.

Keep reading

All terms