vLLM-Omni

The vLLM project's serving framework for models that take or produce more than text: speech, image, video and action models.

  • Serve at scale
  • Apache 2.0 · Open source
  • Linux
  • Verified

vLLM-Omni extends vLLM from text generation to omni-modality models. The README says it handles text, image, audio, video and action data, and it adds support for diffusion transformers and other non-autoregressive models alongside the autoregressive ones that vLLM already serves. It reuses vLLM's KV cache management, overlaps pipeline stages for throughput, can split stages across separate hardware, and exposes an OpenAI-compatible API server with streaming outputs. It also offers full-duplex realtime serving, where audio streams in and out at the same time.

Its supported-model list covers omni models such as Qwen3-Omni and MiniCPM-o 4.5, text-to-speech models including Qwen3-TTS, Tencent AuK, Breeze-TTS-2 and CosyVoice3, image, video and audio generators such as MiniMax H3, LTX-2.5 and Wan2.2, and robot-policy models. Its release 0.32.0 of October 2026, which tracks vLLM 0.32.0, added optional Ray-based multi-node inference and support for YuE2-3B, Qwen-Image-2.1, Kandinsky 6 and MOSS-Image among others. Parallelism options include tensor, pipeline, data and expert parallelism.

According to its installation page it runs on NVIDIA CUDA, AMD ROCm, Intel XPU and Moore Threads MUSA GPUs and on NPUs, and the project publishes nightly CUDA Docker images, deployment recipes and a technical report on arXiv. It is a server-side tool for people deploying models to others. For a single image or voice on your own computer, a runner such as ComfyUI or a model's own script is simpler.

Good for

  • Serving text-to-speech, speech and omni models behind an OpenAI-compatible API
  • Serving image and video diffusion models on GPU servers with vLLM-style batching
  • Realtime voice applications that need streaming audio in and out
  • Teams that already run vLLM and want one runtime for several model types

Watch out for

  • Its releases follow vLLM's version numbers, so pin matching versions and read the release notes for compatibility changes
  • Support differs from model to model, so check the supported-model list and the model's recipe before planning a deployment
  • Setup assumes comfort with Docker, Python environments and GPU drivers

Links