ExLlamaV3
A CUDA inference library for consumer GPUs with its own EXL3 quantization, tensor parallelism and CPU offloading for MoE models.
ExLlamaV3 is an inference library for running local LLMs on modern consumer GPUs. Its features include EXL3 quantization, which is based on QTIP, together with 2 to 8-bit cache quantization, flexible tensor-parallel and expert-parallel inference, and CPU offloading that lets large mixture-of-experts models run with limited GPU memory, with AVX2 and AVX512 support. For generation it offers continuous dynamic batching, speculative decoding and multimodal support.
For an OpenAI-compatible server the README names TabbyAPI as the official and recommended backend, and there is also a plugin for Transformers. Installation needs PyTorch with CUDA 12.4 or later, which means an NVIDIA GPU. Prebuilt wheels are on the releases page, while the PyPI package contains no prebuilt extension and needs the CUDA toolkit and a compiler.
The license is MIT. EXL3 is this library's own weight format and not GGUF, and the README explains how to quantize a model yourself. It is a library, not a chat app, so pair it with TabbyAPI or another front end.
Good for
- Fitting large models onto one or a few NVIDIA consumer GPUs with fine-grained quantization
- Fast local serving through TabbyAPI
- Running big mixture-of-experts models with CPU offload
Watch out for
- It needs an NVIDIA GPU with CUDA 12.4 or later and a matching PyTorch
- The PyPI package builds from source, so use a prebuilt wheel unless you have the CUDA toolkit and a compiler
- EXL3 weights are a separate format from GGUF, so llama.cpp cannot load them