Tools for running models
Models are only files. These are the programs that load them and let you talk to them.
Run locally
Apps and command-line tools that download a model and let you chat with it on your own machine.
- Ollama
The simplest way to download and run open models on your own computer: one command to chat, plus a local API other apps can use.
- llama.cpp
The C/C++ engine behind much of local AI: runs quantized models on almost any hardware, with a command-line tool and an OpenAI-compatible server.
- KoboldCpp
A single-file app that runs GGUF models with a built-in web UI, and also generates images, video, speech and music.
- MLC LLM
A compiler-based engine that deploys the same model on phones, browsers, laptops and GPUs behind an OpenAI-compatible API.
- whisper.cpp
A dependency-free C/C++ port of OpenAI's Whisper that transcribes speech on CPUs, Apple Silicon, GPUs and phones.
- stable-diffusion.cpp
A C/C++ engine, in the style of llama.cpp, that runs image and video diffusion models such as FLUX, Qwen-Image, Z-Image and Wan.
Serve at scale
Engines built for speed and many simultaneous users, usually on GPU servers.
- vLLM
The most widely used open-source engine for serving language models at scale, with an OpenAI-compatible API.
- SGLang
A fast open-source serving framework for language, vision and diffusion models, tuned for agents and large-scale deployment.
- TensorRT LLM
NVIDIA's inference engine for LLMs and visual generation models on NVIDIA GPUs, with optimized kernels and a Python API.
- KTransformers
A research framework that runs very large mixture-of-experts models by sharing the work between a GPU and CPU memory.
- Text Embeddings Inference
Hugging Face's fast server for open text-embedding, re-ranker and classification models, shipped as small Docker images.
- LocalAI
A self-hosted server that puts language, vision, voice, image and video models behind OpenAI-, Anthropic- and ElevenLabs-compatible APIs.
- vLLM-Omni
The vLLM project's serving framework for models that take or produce more than text: speech, image, video and action models.
- LightX2V
A lightweight inference framework for image and video generation, with distilled models, quantization and many hardware back ends.
Libraries
Code libraries that developers build on.
- Hugging Face Transformers
The Python library that defines most open models: the reference way to load, run and fine-tune a model straight from its release.
- MLX LM
Apple's Python package for generating text and fine-tuning language models on Apple silicon Macs with the MLX framework.
- Diffusers
Hugging Face's Python library for running and training diffusion models for images, video and audio.
- ExLlamaV3
A CUDA inference library for consumer GPUs with its own EXL3 quantization, tensor parallelism and CPU offloading for MoE models.
- DiffSynth-Studio
The ModelScope community's open-source diffusion engine for running and training image, video and audio models on modest GPUs.
Interfaces
Web and desktop interfaces that sit on top of a model runner.
- LM Studio
A polished desktop app for downloading, chatting with and serving local models, with an agent and offline voice transcription built in.
- Open WebUI
A self-hosted chat interface that works with Ollama and any OpenAI-compatible API, built to run entirely offline.
- ComfyUI
The node-based studio for open image, video, audio and 3D models: build visual workflows and run them on your own GPU.
- Jan
A free, open-source desktop alternative to ChatGPT that downloads and runs open models offline, and can also connect to cloud APIs.
- AnythingLLM
An all-in-one app for chatting with your documents, building agents and running local models, as a desktop app or a multi-user Docker server.
- InvokeAI
A self-hosted creative studio for AI image generation, with a layered canvas, node-based workflows and a model manager.
Fine-tuning
Tools for adapting a model to your own data.
- Unsloth
Run and fine-tune open models from a desktop app, a web UI or code, with GGUF export and day-one support for many new releases.
- Axolotl
A free fine-tuning framework driven by one YAML file, with LoRA, QLoRA, preference tuning and NVFP4 and expert-parallel training for MoE models.
- LLaMA-Factory
Fine-tune 100+ models with a zero-code command line or web UI, covering LoRA, QLoRA, DPO and multimodal training.
- AI Toolkit
Ostris's all-in-one training suite for image, video and audio diffusion models, with a web interface and a command line.