TensorRT LLM

NVIDIA's inference engine for LLMs and visual generation models on NVIDIA GPUs, with optimized kernels and a Python API.

  • Serve at scale
  • Apache 2.0 · Open source
  • Linux
  • Verified

TensorRT LLM is NVIDIA's open-source toolkit for fast inference on NVIDIA GPUs. According to its README it optimizes LLMs and visual generation models with specialized kernels for common operations, an efficient runtime and a Python framework that you can customize and extend. Its documentation covers installation with pip on Linux, pre-built release containers on NGC and building from source, and it includes a trtllm-serve server with OpenAI-style endpoints.

NVIDIA publishes tuning articles and deployment guides for current open models, for example DeepSeek-V4 on Blackwell, and deployment guides for Nemotron 3 Super, DeepSeek R1 and Qwen3 on Blackwell and Hopper GPUs and for GPT-OSS on Blackwell. The documentation also covers sparse attention, speculative decoding, KV cache offloading and serving with several LoRA adapters.

The license is Apache 2.0, and the license file notes that parts of the code come from other open-source projects with their own licenses. It runs on NVIDIA GPUs only, so check its supported-hardware page for your GPU generation and the deployment guides before you plan a deployment.

Good for

  • Serving open models at high throughput on NVIDIA data-center GPUs
  • Following NVIDIA's own deployment guides for Nemotron, DeepSeek, GPT-OSS and Qwen
  • Production setups on Linux that use NVIDIA's container images

Watch out for

  • It works on NVIDIA GPUs only, and the installation documentation covers Linux
  • The main branch moves ahead of the latest GitHub release, so pin a release or container version
  • Parts of the code carry other open-source licenses, which the license file lists

Links