Self-hosting an open model vs using an API: how to decide

Should you run an open model yourself or call a hosted API? Compare privacy, cost structure, hardware, licenses and maintenance, with a checklist and real memory figures.

  • Intermediate
  • 5 min read
  • Published
  • Updated

Short answer. Self-host an open model when your data must stay under your control, your usage is steady and large, or you need to fine-tune a model or pin its version. Use an API when your volume is low or uneven, when you want the largest models without buying GPUs, or when you need to ship fast. Many teams start on an API, then move the steady part of the workload to open weights, which is exactly what open weights make possible.

Three ways to use a model

  1. A vendor’s API with a closed model. You send requests, you pay per use, and you cannot see or keep the model.
  2. A hosted API for an open model. A provider runs open weights for you. You get per-use pricing and the open model’s license, and the provider handles the hardware.
  3. Self-hosting. You run the model on a PC you own or a GPU server you rent, with a tool such as Ollama or llama.cpp on one machine, or vLLM or SGLang on a server.

The last two are possible only because the weights are open, and only the last puts the files and the hardware under your control.

When self-hosting wins

  • Data control. Prompts and documents never leave your hardware, and you decide where the logs go.
  • National or organizational control. Programs that aim for sovereign AI rely on open weights for the same reason. Our lists of open models made in Europe and made in South Korea show what exists.
  • Fine-tuning. With weights in hand you can fine-tune on your own data and keep the result.
  • A fixed version. Hosted model names can be re-pointed to newer models, so a test you pinned to a name may change. DeepSeek’s old API names, for example, are now served by the new model, as how to evaluate AI models yourself explains. A file on your disk does not change.
  • Offline or private networks. The model runs without an internet connection.
  • Steady, high volume. You pay for hardware instead of per-token fees, which can be cheaper when the GPUs stay busy.

When an API wins

  • Low or uneven volume. GPUs cost the same whether they are busy or idle.
  • The very largest models. Some open models are too big for most budgets, as the next section shows.
  • Speed to launch. An API needs no hardware, drivers or serving stack.
  • No operations team. Self-hosting means you handle updates, scaling, monitoring and failures.
  • Trying many models. An API lets you test a new release in minutes.

What the hardware means in practice

Memory decides what you can run. The table shows the minimum memory by our estimate at 4-bit precision, before the KV cache for long prompts, for representative models. Our VRAM calculator and the guide how much memory do I need give the method.

Class Example Minimum memory (our 4-bit estimate) Typical hardware
Small Gemma 4 E4B about 5.5 GB Phone, laptop or small GPU
Mid gpt-oss-20b about 15 GB One consumer GPU or a Mac
Mid Qwen3.6 27B and 35B-A3B about 19 GB and 25 GB A 24 to 48 GB GPU or a high-memory Mac
Large gpt-oss-120b about 72 GB One 80 GB GPU or several smaller ones
Large Hy3 (295B) about 204 GB A multi-GPU server
Large DeepSeek V4 Flash (284B) about 184 GB A multi-GPU server
Giant DeepSeek V4 Pro (1.6T) about 982 GB A large multi-GPU cluster

Nemotron 3.5 Lightning is built for one 80 GB H100, and our models by VRAM list sorts the whole catalog by the memory you have.

Licenses and hosting clauses

Running an open model inside your own company is allowed by almost every license. The conditions that matter apply in other cases:

AI licenses explained has a checklist to run before you ship.

Think about cost as a formula

Prices change too quickly to quote, so use a formula instead. Compare your monthly server cost, including power or rental, plus the time of whoever maintains it, with the API’s price per token multiplied by the tokens you use per month. Self-hosting breaks even only when the servers stay busy for most of the day. Test with your real traffic pattern, and include peaks, because a server sized for the peak sits idle in quiet hours.

A hybrid path keeps options open

Most of the tools above expose an OpenAI-compatible API, so you can switch between a hosted model and your own by changing a URL and a model name. A sensible path is to prototype on an API, build a test set, compare open models against it, and move the steady workload to your own hardware when the numbers justify it. You can keep sensitive requests on your own server and send the rest to an API. How to choose a runner or inference engine helps you pick the tool, and how to choose an open AI model helps you pick the model.

A checklist

Self-hosting is probably right if most of these are true:

  1. The data must stay on your hardware.
  2. Your traffic is steady rather than spiky.
  3. You need to fine-tune a model or pin its version.
  4. The model you need fits on hardware you can afford.
  5. Someone on your team can run, monitor and update it.
  6. The license allows your use, including any hosting, revenue or territory conditions.
  7. You can accept the time it takes to set up.

If most are false, start with an API, and keep your test set so you can revisit the question when your needs change.

Frequently asked questions

Is self-hosting cheaper than using an API?
Only when you keep the hardware busy. You pay for GPUs whether or not they are working, and you handle updates and scaling yourself. For low or uneven traffic an API usually costs less, and for steady, high volume your own hardware can.
Can I use an open model through an API without hosting it myself?
Yes. Many open models are offered by hosted providers. Check the provider's terms, and note that some licenses restrict businesses that sell hosted access. Some local tools, such as Ollama, can also run models in their own cloud, so confirm where a request is processed.
What hardware do I need to self-host a 30B model?
By our estimate, Qwen3.6 27B needs about 19 GB of memory at 4-bit and the 35B-A3B about 25 GB, plus memory for the KV cache on long prompts. A 24 to 48 GB GPU or a Mac with that much unified memory is the usual target.
Is self-hosting more private?
Your data stays on hardware you control, but you are then responsible for securing it. Some apps can also connect to cloud providers, which would receive your prompts, so check which model is active before typing sensitive text.
Does the license allow me to self-host?
For internal use, almost always. The restrictions that matter most apply to businesses that resell hosted access, and to some conditional licenses with revenue caps or territory limits. Read the license page of your model before you deploy.

Key terms in this guide

  • Inference Inference is the act of running a trained model to get an output, such as answering a prompt, as opposed to training it.
  • Open weights An open-weights model is an AI model whose trained parameters, called weights, are published so that anyone can download and run it, usually under a license that sets the terms of use.
  • OpenAI-compatible API An OpenAI-compatible API is a server that accepts the same requests as the OpenAI API, so apps written for it can use an open model instead.
  • Revenue cap (revenue threshold) A revenue cap is a license condition that makes a model free below a certain income and requires a separate agreement above it.
  • Self-hosting Self-hosting means running a model on hardware you control, such as your own PC or a rented GPU server, instead of calling a vendor's API.
  • Sovereign AI Sovereign AI means AI that a country or organization can build, host and govern under its own control, from the computing power and data to the model itself.

Models mentioned in this guide

Model Released Parameters Context License Smallest variant needs
Qwen3.6 Alibaba Qwen 27.8B – 36B 3B active 256K Apache 2.0 (Permissive) 19 GB Gaming GPU
gpt-oss OpenAI 21B – 117B 5.1B active 128K Apache 2.0 (Permissive) 15 GB Laptop or phone
Gemma 4 Google DeepMind 5.1B – 30.7B 256K Apache 2.0 (Permissive) 3.52 GB Laptop or phone
DeepSeek V4 DeepSeek 284B – 1.6T 49B active 1M MIT (Permissive) 176 GB Big-memory desktop
Hy3 Tencent 295B 21B active 256K Apache 2.0 (Permissive) 204 GB Big-memory desktop
Nemotron 3.5 Lightning NVIDIA 31.6B 3B active 1M OpenMDW-1.1 (Permissive) 22 GB Gaming GPU