Before a local AI can answer questions about your own documents, contracts, or code, something has to convert that text into mathematical vectors it can search through. That something is called an embedding model, and it's the most overlooked piece of building a self-hosted retrieval-augmented generation (RAG) system, which lets you "chat with your own documents" without sending data to the cloud.

A self-hosted RAG system has three layers working together: an embedding model that transforms text into searchable vectors, a vector database that stores and retrieves those vectors, and a local large language model (LLM) that generates answers. Most developers focus on the LLM, the visible part. But the embedding model is where the real work happens, and choosing the wrong one can tank your entire system's accuracy and speed.

What Makes an Embedding Model Right for Your Hardware?

The challenge isn't finding a powerful embedding model; it's finding one that actually runs on your machine. A model that tops every benchmark is useless if it requires more graphics processing power (GPU) memory than you have. The key trade-off is between model size, accuracy, and what your hardware can handle.

For developers starting out with modest hardware, the options are surprisingly practical. The nomic-embed-text model pulls directly from Ollama, a popular tool for running local AI models, and uses only about 0.3 gigabytes of GPU memory while scoring 62.28 on the MTEB benchmark, a standard test for embedding quality. For even tighter constraints, all-MiniLM-L6-v2 runs on CPU alone and uses just 0.1 gigabytes, though with lower accuracy.

If you have more GPU memory available, the Qwen3-Embedding-0.6B model offers what researchers call "best quality-per-VRAM," scoring 70.7 on the MTEB-eng-v2 benchmark while using only 1.5 gigabytes. Scaling up further, the Qwen3-Embedding-8B model delivers even higher accuracy for teams with more powerful hardware.

How to Choose an Embedding Model for Your Self-Hosted Setup

• Assess Your Hardware First: Determine how much GPU or CPU memory you have available. Smaller models like all-MiniLM-L6-v2 (0.1 GB) work on any machine, while larger models like bge-m3 (1.2 GB) need more resources but deliver better accuracy.
• Match the Model to Your Language Needs: If you only work in English, nomic-embed-text or bge-large-en-v1.5 are proven choices. For multilingual documents, bge-m3 or multilingual-e5-large handle 100+ languages and support hybrid dense and sparse retrieval in a single model.
• Verify the License for Commercial Use: Some top-performing models like jina-embeddings-v3 and NV-Embed-v2 use CC-BY-NC licenses that prohibit commercial applications. If you're building a product, stick with Apache-2.0 or MIT-licensed models like nomic-embed-text or bge-m3.
• Check Ollama Compatibility: Models that pull directly from Ollama save setup time. nomic-embed-text, all-minilm, granite-embedding, and bge-m3 are all Ollama-native, meaning you can deploy them with a single command.
• Understand Benchmark Scores Within Context: MTEB scores are only comparable within the same benchmark version. A model scoring 65.8 on MIRACL (a multilingual benchmark) cannot be directly compared to one scoring 62.28 on MTEB-eng-v2 (an English benchmark), even though the numbers look similar.

Which Models Deliver the Best Results in 2026?

The landscape of open-source embedding models has matured significantly. For general-purpose English RAG on modest hardware, nomic-embed-text remains the default choice, supporting documents up to 8,000 tokens long and scoring solidly on standard benchmarks. It's also one of the few models that Ollama supports natively, making deployment straightforward.

For teams needing multilingual support, bge-m3 stands out as a specialist in hybrid retrieval. Unlike most embedding models that only do dense vector search, bge-m3 combines dense retrieval, sparse lexical search, and ColBERT-style multi-vector matching in a single model. This flexibility makes it ideal for RAG systems that need to handle both semantic similarity and exact keyword matches. It supports 8,000-token context windows and works across 100+ languages.

The multilingual-e5-large-instruct model, developed by Microsoft, ranks first among open-source models on the MMTEB multilingual benchmark, which tests performance across 132 different tasks in multiple languages. However, it requires a query instruction prefix to reach peak quality and has a shorter 512-token window, making it better suited for shorter document chunks.

A critical trap awaits developers who chase leaderboard rankings without reading the fine print. NV-Embed-v2 and jina-embeddings-v3 top their respective performance boards, but both carry CC-BY-NC-4.0 licenses that explicitly prohibit commercial use. For any production system, these models are off-limits unless you contact the creators for a commercial license.

Why Benchmark Scores Can Be Misleading?

One of the most confusing aspects of choosing an embedding model is understanding what the numbers actually mean. MTEB scores are not interchangeable across different benchmark versions. The original MTEB benchmark (56 tasks, English only) produces different score ranges than MTEB-eng-v2 or the multilingual MMTEB (132 tasks across multiple languages). A model scoring 65.52 on one version might score very differently on another, even though the numbers look comparable at first glance.

This matters because developers often compare models across benchmarks without realizing they're comparing apples to oranges. The granite-embedding-278m model, for example, scores 48.2 on MTEB English retrieval but 58.3 on the MIRACL multilingual benchmark. These are different tests measuring different things, so the lower number doesn't mean the model is weak; it means it's being measured on a harder, broader task.

The practical takeaway is to read the benchmark name in each model's specification, not just the number. If you're building an English-only system, compare models only on MTEB-eng-v2 scores. If you need multilingual support, use MMTEB or MIRACL scores as your guide.

The Emerging Trend: Mixture-of-Experts Embedding Models

A newer category of embedding models is gaining traction: mixture-of-experts (MoE) designs that activate only part of the model during inference. The nomic-embed-text-v2-moe model, for instance, has 475 million total parameters but only activates 305 million during use, cutting computational cost while maintaining quality. It scores 65.8 on the MIRACL multilingual benchmark and 52.86 on BEIR, a retrieval benchmark, while using just 1 gigabyte of memory.

This approach is significant because it lets developers run larger, more capable models on hardware that would otherwise be too constrained. As local AI adoption grows, MoE embedding models may become the standard way to balance accuracy and efficiency.

The embedding model layer remains the most under-discussed part of self-hosted RAG, but it's also where the most practical gains can be made. Choosing the right model for your hardware, language needs, and license requirements can mean the difference between a RAG system that works reliably and one that frustrates users with poor search results.