How to choose an open-source embedding model for search and RAG

Pick an open embedding model by language, input length, modality, size and license, and compare BGE-M3, Qwen3-Embedding, EmbeddingGemma 2, Granite Embedding R2 and more.

  • Intermediate
  • 7 min read
  • Published
  • Updated

Short answer. Choose the embedding model by what you search, not by its leaderboard rank. BGE-M3 (MIT) and Qwen3-Embedding (Apache 2.0) are safe defaults for multilingual text, Granite Embedding R2 suits CPUs and small servers, and EmbeddingGemma 2 or Qwen3-VL-Embedding handle images and video, and Nemotron 3.5 Embed searches page images of documents with text queries. Check the license, test two or three models on a few hundred real queries, and remember that changing the model later means re-indexing everything.

What an embedding model does

An embedding model turns text, and in newer models images, video or audio, into a list of numbers so that similar items end up close together. You embed your documents once, embed each question, and return the closest documents. It is the retrieval half of retrieval-augmented generation.

The choice matters more than its size suggests, because it is hard to undo. Vectors from different models live in different spaces, so a new model needs a full re-index of your data. Choose with care now, and look at the embedding model list for every model we track.

The shortlist

Model Size License Input limit Covers
BGE-M3 568M MIT 8,192 tokens 100+ languages; dense, sparse and multi-vector search
Qwen3-Embedding 0.6B, 4B, 8B Apache 2.0 32K tokens 100+ languages and code; adjustable vector length
EmbeddingGemma 2 270M to 740M Apache 2.0 8,192 tokens Text, code, images, video and audio in one space
Qwen3-VL-Embedding 2B, 8B Apache 2.0 32K tokens Text, images, screenshots and video; 30+ languages
Nemotron 3.5 Embed 1B, 8B OpenMDW-1.1 32K tokens Text queries over text and document-page images; a matching 8B reranker
Granite Embedding R2 47M to 311M Apache 2.0 8K or 32K tokens English, or 52 enhanced languages plus code
voyage-4-nano 346M Apache 2.0 32,000 tokens (card) Multilingual; vectors shared with the hosted Voyage 4 models
Harrier OSS v1 270M, 0.6B, 27B MIT 32,768 tokens 42 named languages; Microsoft’s small and large sizes
pplx-embed 0.6B to 9B MIT (one card states none) 32K tokens (v1) Dense, contextual-chunk and late-interaction models; v2 also takes images
Giga-Embeddings 480M, 3B, 10B-A1.8B MIT 8,192 positions (config) Russian and English; Sber’s own MTEB Russian scores
Jina Embeddings v5 239M, 677M CC BY-NC 8,192 or 32,768 tokens 119+ languages with the small model
Jina Embeddings v5 Omni 1B nano, 1.6B small CC BY-NC 8,192 or 32,768 tokens Text, images, video and audio in one space; text vectors match v5 text

Scores on benchmarks such as MTEB are mostly reported by the developers and are easy to tune for, so we leave them out of the table.

Five questions that decide the choice

  1. What are you searching? Text and code only, or also images, screenshots, video and audio? For pictures and mixed documents, use EmbeddingGemma 2 or Qwen3-VL-Embedding. For scans, slides and reports with tables and charts, Nemotron 3.5 Embed searches the page images directly, so you can skip OCR, though its queries must be text. pplx-embed v2 also takes images with late-interaction scoring that stores one small vector per token, and Jina Embeddings v5 Omni puts text, images, video and audio in one space. For text, the text-only models are lighter.
  2. Which languages? BGE-M3 and Qwen3-Embedding list 100+ languages. Granite Embedding R2 is best on its 52 enhanced languages and code, and relies on the encoder’s pretraining for others. Qwen3-VL-Embedding covers fewer, at 30+, and the Nemotron 3.5 Embed cards claim evaluation in 34 languages for text and 18 for images. Harrier OSS v1 names 42 languages, and Giga-Embeddings documents Russian and English only, so it is the one to test first for Russian.
  3. How long are your passages? Qwen3-Embedding, Qwen3-VL-Embedding, Nemotron 3.5 Embed, Harrier OSS v1, pplx-embed v1 and the multilingual Granite models read up to about 32K tokens. BGE-M3, EmbeddingGemma 2 and Giga-Embeddings (by its configuration) stop at 8,192, so split long documents into chunks.
  4. Where will it run? A phone or a CPU-only server points to EmbeddingGemma 2, the small Granite models or Harrier’s 270M model, whose BF16 file is 0.54 GB. A GPU server can run the 4B and 8B Qwen models when you want higher accuracy, while Harrier’s 27B model is a 54 GB file.
  5. What does the license allow? Jina Embeddings v5 is distilled from Qwen3-Embedding-4B and released under CC BY-NC, and the Omni models carry the same license, so both rule out commercial products unless you get a separate license or use Elastic’s hosted service. AI licenses explained covers the other terms.

Add a reranker for the last step

Search with embeddings is fast but can miss subtle matches. A reranker reads the question and each candidate together and re-scores a short list. The usual recipe is to fetch the top 50 or 100 candidates with the embedding model and let the reranker sort them. A reranker is a separate model with its own license, and it adds delay to every query, so keep the candidate list short. The reranker list covers every one we track, and these are the main choices:

Model Size License Input Notes
Qwen3-Reranker 0.6B, 4B, 8B Apache 2.0 Text, 32K tokens 100+ languages and code; follows instructions
BGE Reranker v2 m3 568M Apache 2.0 Text, 8K positions (config) Multilingual; about 14.7 million downloads in 30 days
Jina Reranker v3.5 0.6B CC BY-NC Text, up to 131K tokens (card) Scores a whole candidate list in one pass
Qwen3-VL-Reranker 2B, 8B Apache 2.0 Text, images, screenshots and video, 32K tokens Follows Qwen3-VL-Embedding
Nemotron 3.5 Rerank 8B OpenMDW-1.1 Text and page images, 32K tokens Follows Nemotron 3.5 Embed

A reranker sees only the query and each candidate, not the first-stage vectors, so you can pair any of them with any embedding model, for example BGE-M3 for retrieval and Qwen3-Reranker for sorting. Test the pair on your own queries, with and without the reranker, because every score published for these models comes from the developers.

BGE-M3 offers another route to better recall inside one model: it returns dense, sparse and multi-vector scores, which you can combine in hybrid search.

Shrink the vectors

Vector storage grows with the number of dimensions. One million chunks at 1,024 dimensions in 32-bit floats take about 4 GB before any index overhead, and at 256 dimensions they take about 1 GB. Matryoshka embeddings are trained so the first numbers still work on their own, which lets you cut the vector shorter. EmbeddingGemma 2 can shorten its 768 dimensions to 512, 256 or 128, Qwen3-Embedding lets you choose the vector length, Nemotron 3.5 Embed can cut its vectors to 1,024 or 512 dimensions and renormalize them, pplx-embed supports Matryoshka truncation and gives native int8 vectors with a binary option, voyage-4-nano goes from 2,048 down to 256 dimensions with int8 or binary output, and Jina Embeddings v5 describes Matryoshka truncation and binary quantization. Use only the lengths the model was trained for, and test retrieval quality after you truncate.

Follow the model’s input format

Embedding models often expect a specific input format, and ignoring it lowers quality:

  • Qwen3-Embedding is instruction-aware. You prepend a short task description to the query, and Qwen recommends writing it in English even for other languages.
  • EmbeddingGemma 2 works best with its recommended task prefixes for text, while images, video and audio take none.
  • Harrier OSS v1 and Giga-Embeddings both want an “Instruct: … Query: …” prefix on queries but not on documents. Harrier takes the last token as the vector, while Giga-Embeddings needs mean pooling, so copy the pooling from each card.
  • voyage-4-nano uses a different prompt for queries and for documents, and Jina Embeddings v5 Omni uses “Query: “ and “Document: “ prefixes for every kind of input.
  • pplx-embed v1 needs no prefix, and its int8 vectors are unnormalized, so compare them with cosine similarity.
  • Jina Embeddings v5 ships four task adapters, for retrieval, text matching, clustering and classification, so you pick the one that fits.

Keep each chunk within the model’s input limit. Splitting at headings or paragraphs keeps meaning together, and an OCR model that writes Markdown makes that easier for scanned documents.

Test on your own queries

Collect 100 to 300 real questions, and for each one note the passage that should be found. Run each candidate and measure how often the right passage appears in the top 5. Compare two or three models, with and without a reranker, and look at the failures by hand. How to evaluate AI models yourself covers the method in more detail.

When you are ready to serve a model, Text Embeddings Inference runs open embedding and re-ranker models behind an HTTP API, and it serves text models only. The Nemotron 3.5 Embed cards show usage with Transformers, Sentence Transformers and vLLM. Keep the model name and version next to your index, so that when a better model arrives, you know exactly what you must rebuild.

Frequently asked questions

Which open embedding model is the most popular?
BGE-M3 had about 29 million downloads on Hugging Face in the 30 days to 11 November 2026, far more than any other embedding model we list. Popularity shows that a model is easy to use, not that it is the best for your documents.
Can I mix vectors from two different embedding models?
No. Each model builds its own vector space, so vectors from different models, or even from different sizes of one family, cannot be compared. Switching models means embedding all of your documents again.
Do I need a GPU for embeddings?
Usually not for the small models. Granite Embedding R2 starts at 47M parameters and is small enough for a CPU, and EmbeddingGemma 2 is built to run on-device. By our estimate, a 0.6B model needs about 1.4 GB of memory in BF16.
Can I use these embedding models in a commercial product?
Most can. BGE-M3, Harrier OSS v1 and Giga-Embeddings are MIT, and so are the pplx-embed cards we checked, except one that states no license. Qwen3-Embedding, EmbeddingGemma 2, Qwen3-VL-Embedding, Granite Embedding R2 and voyage-4-nano are Apache 2.0. Nemotron 3.5 Embed uses the permissive OpenMDW-1.1. Jina Embeddings v5 and v5 Omni are CC BY-NC, so they are free for non-commercial use only.
Do I need a reranker as well?
It is optional. A reranker re-scores the top results by reading the question and each result together, which is more accurate but adds delay to every query. Consider adding one when the first-stage results miss passages that should have been found. Qwen3-Reranker and BGE Reranker v2 m3 are Apache 2.0, Nemotron 3.5 Rerank uses OpenMDW-1.1, and Jina Reranker v3.5 is non-commercial.

Key terms in this guide

  • Embedding model An embedding model turns text, images or other data into a list of numbers that captures its meaning, so that similar items end up close together.
  • Matryoshka embeddings Matryoshka embeddings are trained so that the first part of the vector still works on its own, which lets you cut the vector shorter to save storage and search time.
  • MTEB (embedding benchmark) MTEB, the Massive Text Embedding Benchmark, scores embedding models on many tasks such as search, classification and clustering, and its public leaderboard is a common way to compare them.
  • Retrieval-augmented generation (RAG) Retrieval-augmented generation lets a model answer from your own documents by retrieving the relevant passages and adding them to the prompt.
  • Reranker A reranker re-scores a short list of search results by reading the question and each result together, which is slower but more accurate than embeddings alone.

Models mentioned in this guide

Model Released Parameters Context License Smallest variant needs
BGE-M3 BAAI 568M 8K MIT (Permissive) 2.61 GB Laptop or phone
Qwen3-Embedding Alibaba Qwen 600M – 7.57B 32K Apache 2.0 (Permissive) 1.38 GB Laptop or phone
EmbeddingGemma 2 Google DeepMind 270M – 740M 8K Apache 2.0 (Permissive) 0.621 GB Laptop or phone
Qwen3-VL-Embedding Alibaba Qwen 2.13B – 8.14B 32K Apache 2.0 (Permissive) 4.9 GB Laptop or phone
Nemotron 3.5 Embed NVIDIA 1.55B – 8.38B 32K OpenMDW-1.1 (Permissive) 1.46 GB Laptop or phone
Granite Embedding R2 IBM 48M – 312M 32K Apache 2.0 (Permissive) 0.115 GB Laptop or phone
Harrier OSS v1 Microsoft 270M – 27B 32K MIT (Permissive) 0.621 GB Laptop or phone
pplx-embed Perplexity 590M – 8.4B 32K MIT (Permissive) 2.76 GB Laptop or phone
Giga-Embeddings Sber (GigaChat) 480M – 10.5B 1.8B active 8K MIT (Permissive) 1.13 GB Laptop or phone
voyage-4-nano Voyage AI 346M 32K Apache 2.0 (Permissive) 0.425 GB Laptop or phone
Jina Embeddings v5 Jina AI 239M – 677M 32K CC BY-NC (Non-commercial) 0.552 GB Laptop or phone
Jina Embeddings v5 Omni Jina AI 990M – 1.63B 32K CC BY-NC (Non-commercial) 2.27 GB Laptop or phone
Qwen3-Reranker Alibaba Qwen 596M – 8.19B 32K Apache 2.0 (Permissive) 1.37 GB Laptop or phone
BGE Reranker v2 m3 BAAI 568M 8K Apache 2.0 (Permissive) 2.61 GB Laptop or phone
Jina Reranker v3.5 Jina AI 597M 128K CC BY-NC (Non-commercial) 0.46 GB Laptop or phone
Qwen3-VL-Reranker Alibaba Qwen 2.13B – 8.77B 32K Apache 2.0 (Permissive) 4.9 GB Laptop or phone
Nemotron 3.5 Rerank 8B NVIDIA 8.38B 32K OpenMDW-1.1 (Permissive) 19 GB Gaming GPU