Self-hosting

Self-hosting means running a model on hardware you control, such as your own PC or a rented GPU server, instead of calling a vendor's API.

  • Using models
  • Updated

Open weights make self-hosting possible: you download the model and serve it yourself, with a tool such as Ollama or llama.cpp on one machine, or vLLM or SGLang on a server. You decide where the data goes and you can fine-tune the model, and you pay for hardware instead of per-token fees.

It is not always cheaper or easier. You pay for GPUs whether or not they are busy, you handle updates and scaling, and the largest models need many GPUs. Many open models can also be used through hosted APIs, and some licenses restrict selling hosted access, so check the license if you plan to offer a model as a service.

Related terms

  • Inference Inference is the act of running a trained model to get an output, such as answering a prompt, as opposed to training it.
  • OpenAI-compatible API An OpenAI-compatible API is a server that accepts the same requests as the OpenAI API, so apps written for it can use an open model instead.
  • VRAM VRAM is the memory on a graphics card; it decides which models a GPU can hold, because the weights must fit in memory for the model to run quickly.
  • Open weights An open-weights model is an AI model whose trained parameters, called weights, are published so that anyone can download and run it, usually under a license that sets the terms of use.

Keep reading

All terms