SGLang
A fast open-source serving framework for language, vision and diffusion models, tuned for agents and large-scale deployment.
SGLang is an inference framework for large language models, vision-language models and diffusion models. Its maintainers say it is optimized for agentic workloads, large-scale serving and reinforcement-learning rollouts. It includes SGLang Diffusion, a built-in engine for image and video generation that ships in the same Python package.
It is often used as an alternative to vLLM, and many new open models, such as FLUX 3 Image Klein, list it among their day-one supported runtimes. Its documentation offers a cookbook where you pick a model and your hardware and get a ready-to-run launch command, which is very helpful for large mixture-of-experts models.
Supported hardware is broad: NVIDIA GPUs from RTX cards to B200 and GB300, AMD Instinct, Google TPUs, Intel GPUs and Xeon CPUs, Huawei Ascend NPUs, Moore Threads GPUs, and Apple silicon Macs through Metal and MLX. You can run it from a Docker image or install it with uv. Like vLLM, it is meant for people deploying models for others, not for a first local experiment.
Good for
- Serving large mixture-of-experts models for many users
- Agent and tool-calling workloads that reuse long prompts
- Generating images and video with a server-side engine
- Teams that want a second option next to vLLM
Watch out for
- Setup assumes you are comfortable with Docker, Python environments and GPU drivers
- Launch flags depend on the model and hardware, so follow the model card and cookbook
- It moves quickly, so pin a version for production