Text Embeddings Inference

Hugging Face's fast server for open text-embedding, re-ranker and classification models, shipped as small Docker images.

  • Serve at scale
  • Apache 2.0 · Open source
  • macOS, Linux
  • Verified

Text Embeddings Inference (TEI) is a toolkit from Hugging Face for deploying and serving open-source text embedding and sequence-classification models. The README says it supports popular models including FlagEmbedding, Ember, GTE and E5, and it also covers re-ranker models, SPLADE pooling and models with private or gated access. Features include token-based dynamic batching, small Docker images with fast boot times, optimized kernels based on Flash Attention, safetensors and ONNX weight loading, and production tooling such as Prometheus metrics and OpenTelemetry tracing.

It needs no model-graph compilation step, has Metal support for local use on Macs through a Homebrew install, and has a build for AMD Instinct GPUs. It exposes an HTTP API, with gRPC as an option, that a RAG pipeline can call to embed text and to re-rank results. Check the supported-models list in the README before you choose an embedding model, because only some architectures are supported.

The license is Apache 2.0. It serves text models only, so pair it with an embedding model from our embedding list and a vector database.

Good for

  • Serving an embedding or re-ranker model behind an API for RAG
  • High-throughput embedding with dynamic batching
  • Small container deployments with fast start-up
  • Local use on a Mac through Metal

Watch out for

  • It serves text models only, not image or audio embedding models
  • Only the architectures in its supported-models list work, so check the list first
  • Docker images are built per hardware type, so pick the tag that matches your GPU

Links