MTEB (embedding benchmark)

MTEB, the Massive Text Embedding Benchmark, scores embedding models on many tasks such as search, classification and clustering, and its public leaderboard is a common way to compare them.

  • Using models
  • Updated

MTEB was introduced in an October 2022 paper. The original version spanned 8 task types, 58 datasets and 112 languages, and the benchmark has been extended since. A model gets a score per task and an average, and the open-source code and public leaderboard let anyone compare models.

Scores in model cards are usually self-reported, and a high average does not make a model the best for your data. Models can be tuned toward public test sets, and your documents may be in another language or domain. Build a small test set of real questions and the passages that answer them, and measure how often each model finds the right passage.

Related terms

  • Embedding model An embedding model turns text, images or other data into a list of numbers that captures its meaning, so that similar items end up close together.
  • Benchmark A benchmark is a standardized test used to compare models, such as a set of maths, coding or knowledge questions; scores are useful but easy to over-read.
  • Retrieval-augmented generation (RAG) Retrieval-augmented generation lets a model answer from your own documents by retrieving the relevant passages and adding them to the prompt.

Keep reading

All terms