OpenAI-compatible API

An OpenAI-compatible API is a server that accepts the same requests as the OpenAI API, so apps written for it can use an open model instead.

  • Using models
  • Updated

llama.cpp and vLLM both offer such a server, and front ends such as Open WebUI can connect to any server of this kind. To switch an application to a local model you usually change only the base URL and the model name, and leave the code that builds the request as it is.

Compatibility is partial. The basics, such as chat completions, work almost everywhere, but features like tool calling, structured output and image input vary by server and by model, so test the ones you rely on.

Related terms

  • Inference Inference is the act of running a trained model to get an output, such as answering a prompt, as opposed to training it.

Keep reading

All terms