vLLM
The most widely used open-source engine for serving language models at scale, with an OpenAI-compatible API.
vLLM is an inference and serving engine built for throughput. It started in the Sky Computing Lab at UC Berkeley and is now maintained by a large community. Its signature idea is PagedAttention, which manages the memory used for attention key and value caches efficiently, combined with continuous batching of incoming requests, chunked prefill and prefix caching. Together these let one GPU server answer many users at once.
It supports a broad range of quantization formats, including FP8, MXFP4, NVFP4, INT8, INT4, GPTQ, AWQ and GGUF; speculative decoding; structured outputs; tool calling; and multi-LoRA serving. For large models it can split work across GPUs with tensor, pipeline, data, expert and context parallelism. It exposes an OpenAI-compatible API server, plus the Anthropic Messages API and gRPC, and works with Hugging Face models. According to the project, it supports more than 200 model architectures.
It runs on NVIDIA, AMD and Intel GPUs and on several CPU types, and hardware plugins add Google TPUs, Intel Gaudi, Huawei Ascend and others. It is mainly a Linux, server-side tool: choose it when you are deploying a model for a team or product, not for casual use on a laptop.
Good for
- Serving an open model to many users with high throughput
- Multi-GPU and multi-node deployment of very large models
- Replacing a paid model API with a self-hosted, OpenAI-compatible endpoint
- Teams that already run Linux servers with GPUs
Watch out for
- It is built for GPU servers, and setup is heavier than Ollama or LM Studio
- New model architectures can need a recent vLLM version, so check the model card for supported versions
- Memory planning still matters, because the KV cache competes with the weights for GPU memory