How to choose a runner or inference engine for open models
Ollama, LM Studio, llama.cpp, MLX-LM, vLLM, SGLang, TensorRT-LLM and KTransformers compared: which tool fits a laptop, a Mac, a phone, a home server or a production API.
Short answer. On one computer, start with Ollama or LM Studio, and move to llama.cpp when you want control. On a Mac, add MLX-LM. To serve many users on GPU servers, use vLLM or SGLang, plus TensorRT-LLM on NVIDIA data-center GPUs. To fit a giant mixture-of-experts model on one GPU with a lot of system memory, try KTransformers. Check that the engine supports your model before you download the weights.
Start from where it will run
The right tool depends on where the model runs and how many people use it:
- Your own computer, for one person. Pick a runner that downloads a model and answers questions in a few minutes.
- An app, a browser or a phone. Pick an engine you can embed in a product.
- A server, for many users. Pick an engine built for batching, long contexts and several GPUs.
Inference is the step of running a trained model, and the OpenAI-compatible API that most of these tools expose means you can change your mind later without rewriting your application.
The tools at a glance
| Tool | Type | License | Runs on | Use it when |
|---|---|---|---|---|
| Ollama | Runner | MIT | macOS, Windows, Linux | You want one command to chat and a local API |
| LM Studio | Desktop app | Proprietary, free plan | macOS, Windows, Linux | You prefer a graphical app; the app is not open source |
| llama.cpp | Runner | MIT | macOS, Windows, Linux | You want 2-bit to 6-bit quantization and control on almost any hardware |
| KoboldCpp | Runner | AGPL-3.0 | macOS, Windows, Linux | You want one file with a built-in web UI |
| MLX-LM | Library | MIT | macOS | You have an Apple silicon Mac and want Python and LoRA |
| MLC LLM | Runner | Apache 2.0 | macOS, Windows, Linux | You ship a model in an iOS, Android or browser app |
| ExLlamaV3 | Library | MIT | Windows, Linux | You use NVIDIA consumer GPUs and want EXL3 quantization |
| Transformers | Library | Apache 2.0 | macOS, Windows, Linux | You want to try a model the day it ships, in Python |
| vLLM | Server | Apache 2.0 | Linux | You serve many users on GPU servers |
| SGLang | Server | Apache 2.0 | Linux, macOS | You serve large mixture-of-experts or agent workloads |
| TensorRT-LLM | Server | Apache 2.0 | Linux | You run NVIDIA GPUs and follow NVIDIA’s deployment guides |
| KTransformers | Server | Apache 2.0 | Linux | You run a giant mixture-of-experts model on one GPU plus system RAM |
On your own computer
Beginners. LM Studio and Ollama both get you chatting quickly. LM Studio is a polished desktop app that can also serve a local API, but it is proprietary, so read its terms before work use, and it does not support Intel Macs. Ollama is a command-line runner with a library of ready models and a local API. Its library models are repackaged and often quantized, and it can also run models in its cloud, so confirm that a model runs locally before you send sensitive data.
More control. llama.cpp runs GGUF models on CPUs, older GPUs, Apple silicon and unusual hardware, and it includes an OpenAI-compatible server. Support for a brand-new architecture can lag behind a release. KoboldCpp wraps the same kind of models in a single downloadable file with a built-in interface and also generates images and speech.
Apple silicon. MLX-LM is Apple’s package for text generation and LoRA fine-tuning on Macs, and it runs on Apple silicon only.
NVIDIA consumer GPUs. ExLlamaV3 offers its own EXL3 quantization, which llama.cpp cannot load, and CPU offloading for large mixture-of-experts models. It needs CUDA 12.4 or later.
A friendly front end. Open WebUI, Jan and AnythingLLM add a chat interface, document search or agents; Open WebUI needs a model server such as Ollama underneath. The guide to running AI models locally walks through a first setup.
Inside an app or on a phone
MLC LLM compiles the same model for iOS, Android, browsers, laptops and GPUs behind an OpenAI-compatible API. Models must be converted and compiled for each target, so confirm that your model is supported first. llama.cpp is the other common choice for on-device work, and the glossary entry on on-device AI explains the trade-offs.
Serving many users
Server engines batch many requests together and manage the KV cache, which competes with the weights for GPU memory. All three below run on Linux:
- vLLM is the most widely used open-source engine for serving language models. It handles multi-GPU and multi-node deployment and gives you a self-hosted, OpenAI-compatible endpoint. New architectures may need a recent version.
- SGLang is tuned for large mixture-of-experts models and for agent workloads that reuse long prompts. It can also serve image and video models. It moves quickly, so pin a version for production.
- TensorRT-LLM is NVIDIA’s engine for its own GPUs, with optimized kernels and deployment guides for models such as Nemotron, DeepSeek, GPT-OSS and Qwen. The main branch moves ahead of the latest release, so pin a release or a container.
Launch flags depend on the model and the hardware, so start from the recipe on the model card, such as the one for GLM-5.3 Flash, which Z.ai lists for SGLang and vLLM.
Giant models on modest hardware
Mixture-of-experts models activate only a few experts per token, so the experts that are idle can sit in system memory. KTransformers shares the work between a GPU and CPU memory, so giants from DeepSeek, GLM, Kimi or MiniMax can run on one GPU with a lot of RAM, at a lower speed than on all-GPU servers. It is a research project, and its packages need Linux on x86-64. ExLlamaV3 and llama.cpp also support CPU offload for such models. Before you try, read how much memory do I need and look at models that ship in 4-bit, as described in giant open models ship in 4-bit.
Check support before you download
Three checks save hours:
- Does the engine support the model’s architecture? Read the model card for the recommended engine and version. A model can run in Transformers on day one and take longer to reach llama.cpp.
- Which file format? Safetensors weights are what Transformers, vLLM and SGLang load, GGUF is the format of llama.cpp and many local apps, and EXL3 works only with ExLlamaV3. Quantization explained covers the formats.
- Does it fit? Add the KV cache for your longest prompts to the weights, and test your own task at the precision you will actually use.
If you are still choosing a model, how to choose an open AI model is the place to start.
Frequently asked questions
- Should I use Ollama or llama.cpp?
- Use Ollama if you want the simplest start, with one command to chat and a local API. Use llama.cpp if you want more control over quantization, hardware and server options, and you are comfortable with the command line. Ollama's library models are repackaged and often quantized, so check the tag before you compare quality with the original release.
- What is the difference between vLLM and SGLang?
- Both are open-source engines that serve models on GPU servers behind an OpenAI-compatible API. vLLM is the most widely used, and SGLang is tuned for large mixture-of-experts models and for agent workloads that reuse long prompts. Follow the launch recipe on the model card, and test both on your own traffic if you can.
- Can I run open models on a Mac?
- Yes. LM Studio, Ollama and llama.cpp run on Apple silicon, and MLX-LM is Apple's own package for text generation and LoRA fine-tuning on Apple silicon only. Large models need a lot of unified memory, so check the memory guide first.
- Do I need a server engine to serve many users?
- For many users at once, yes. Ollama is aimed at local and small-team use, while vLLM, SGLang and TensorRT-LLM are built for batching many requests on GPU servers. They also take more setup.
- Why does an OpenAI-compatible API matter?
- Most of these tools expose the same style of API, so your application can switch engines, or switch between a local model and a hosted one, by changing a URL and a model name instead of rewriting code.
Key terms in this guide
- GGUF GGUF is a single-file model format used by llama.cpp and the tools built on it, which bundles quantized weights with the information needed to run them.
- Inference Inference is the act of running a trained model to get an output, such as answering a prompt, as opposed to training it.
- KV cache The KV cache is the memory a model uses to remember the earlier tokens of a conversation, and it grows with the length of the context.
- OpenAI-compatible API An OpenAI-compatible API is a server that accepts the same requests as the OpenAI API, so apps written for it can use an open model instead.
- Self-hosting Self-hosting means running a model on hardware you control, such as your own PC or a rented GPU server, instead of calling a vendor's API.
Models mentioned in this guide
| Model | Released | Parameters | Context | License | Smallest variant needs |
|---|---|---|---|---|---|
| Qwen3.6 Alibaba Qwen | 27.8B – 36B 3B active | 256K | Apache 2.0 (Permissive) | 19 GB Gaming GPU | |
| Gemma 4 Google DeepMind | 5.1B – 30.7B | 256K | Apache 2.0 (Permissive) | 3.52 GB Laptop or phone | |
| gpt-oss OpenAI | 21B – 117B 5.1B active | 128K | Apache 2.0 (Permissive) | 15 GB Laptop or phone | |
| DeepSeek V4 DeepSeek | 284B – 1.6T 49B active | 1M | MIT (Permissive) | 176 GB Big-memory desktop | |
| GLM-5.3-Flash Z.ai (Zhipu AI) | 320B 18B active | 1M | MIT (Permissive) | 221 GB Big-memory desktop | |
| Nemotron 3.5 Lightning NVIDIA | 31.6B 3B active | 1M | OpenMDW-1.1 (Permissive) | 22 GB Gaming GPU |