Tools for running models

Models are only files. These are the programs that load them and let you talk to them.

Run locally

Apps and command-line tools that download a model and let you chat with it on your own machine.

  • Ollama

    The simplest way to download and run open models on your own computer: one command to chat, plus a local API other apps can use.

    Open sourceMITmacOS, Windows, Linux
  • llama.cpp

    The C/C++ engine behind much of local AI: runs quantized models on almost any hardware, with a command-line tool and an OpenAI-compatible server.

    Open sourceMITmacOS, Windows, Linux
  • KoboldCpp

    A single-file app that runs GGUF models with a built-in web UI, and also generates images, video, speech and music.

    Open sourceAGPL-3.0macOS, Windows, Linux
  • MLC LLM

    A compiler-based engine that deploys the same model on phones, browsers, laptops and GPUs behind an OpenAI-compatible API.

    Open sourceApache 2.0macOS, Windows, Linux
  • whisper.cpp

    A dependency-free C/C++ port of OpenAI's Whisper that transcribes speech on CPUs, Apple Silicon, GPUs and phones.

    Open sourceMITmacOS, Windows, Linux
  • stable-diffusion.cpp

    A C/C++ engine, in the style of llama.cpp, that runs image and video diffusion models such as FLUX, Qwen-Image, Z-Image and Wan.

    Open sourceMITmacOS, Windows, Linux

Serve at scale

Engines built for speed and many simultaneous users, usually on GPU servers.

  • vLLM

    The most widely used open-source engine for serving language models at scale, with an OpenAI-compatible API.

    Open sourceApache 2.0Linux
  • SGLang

    A fast open-source serving framework for language, vision and diffusion models, tuned for agents and large-scale deployment.

    Open sourceApache 2.0Linux, macOS
  • TensorRT LLM

    NVIDIA's inference engine for LLMs and visual generation models on NVIDIA GPUs, with optimized kernels and a Python API.

    Open sourceApache 2.0Linux
  • KTransformers

    A research framework that runs very large mixture-of-experts models by sharing the work between a GPU and CPU memory.

    Open sourceApache 2.0Linux
  • Text Embeddings Inference

    Hugging Face's fast server for open text-embedding, re-ranker and classification models, shipped as small Docker images.

    Open sourceApache 2.0macOS, Linux
  • LocalAI

    A self-hosted server that puts language, vision, voice, image and video models behind OpenAI-, Anthropic- and ElevenLabs-compatible APIs.

    Open sourceMITmacOS, Linux
  • vLLM-Omni

    The vLLM project's serving framework for models that take or produce more than text: speech, image, video and action models.

    Open sourceApache 2.0Linux
  • LightX2V

    A lightweight inference framework for image and video generation, with distilled models, quantization and many hardware back ends.

    Open sourceApache 2.0Linux

Libraries

Code libraries that developers build on.

  • Hugging Face Transformers

    The Python library that defines most open models: the reference way to load, run and fine-tune a model straight from its release.

    Open sourceApache 2.0macOS, Windows, Linux
  • MLX LM

    Apple's Python package for generating text and fine-tuning language models on Apple silicon Macs with the MLX framework.

    Open sourceMITmacOS
  • Diffusers

    Hugging Face's Python library for running and training diffusion models for images, video and audio.

    Open sourceApache 2.0macOS, Windows, Linux
  • ExLlamaV3

    A CUDA inference library for consumer GPUs with its own EXL3 quantization, tensor parallelism and CPU offloading for MoE models.

    Open sourceMITWindows, Linux
  • DiffSynth-Studio

    The ModelScope community's open-source diffusion engine for running and training image, video and audio models on modest GPUs.

    Open sourceApache 2.0

Interfaces

Web and desktop interfaces that sit on top of a model runner.

  • LM Studio

    A polished desktop app for downloading, chatting with and serving local models, with an agent and offline voice transcription built in.

    Free to use, not open sourceProprietary (free plan)macOS, Windows, Linux
  • Open WebUI

    A self-hosted chat interface that works with Ollama and any OpenAI-compatible API, built to run entirely offline.

    Free to use, not open sourceOpen WebUI LicensemacOS, Windows, Linux
  • ComfyUI

    The node-based studio for open image, video, audio and 3D models: build visual workflows and run them on your own GPU.

    Open sourceGPL-3.0macOS, Windows, Linux
  • Jan

    A free, open-source desktop alternative to ChatGPT that downloads and runs open models offline, and can also connect to cloud APIs.

    Open sourceApache 2.0macOS, Windows, Linux
  • AnythingLLM

    An all-in-one app for chatting with your documents, building agents and running local models, as a desktop app or a multi-user Docker server.

    Open sourceMITmacOS, Windows, Linux
  • InvokeAI

    A self-hosted creative studio for AI image generation, with a layered canvas, node-based workflows and a model manager.

    Open sourceApache 2.0macOS, Windows, Linux

Fine-tuning

Tools for adapting a model to your own data.

  • Unsloth

    Run and fine-tune open models from a desktop app, a web UI or code, with GGUF export and day-one support for many new releases.

    Open sourceApache 2.0 and AGPL-3.0macOS, Windows, Linux
  • Axolotl

    A free fine-tuning framework driven by one YAML file, with LoRA, QLoRA, preference tuning and NVFP4 and expert-parallel training for MoE models.

    Open sourceApache 2.0Linux
  • LLaMA-Factory

    Fine-tune 100+ models with a zero-code command line or web UI, covering LoRA, QLoRA, DPO and multimodal training.

    Open sourceApache 2.0Windows, Linux
  • AI Toolkit

    Ostris's all-in-one training suite for image, video and audio diffusion models, with a web interface and a command line.

    Open sourceMITmacOS, Windows, Linux