10 ms·

Gemini Embedding: Powering RAG and context engineering

278 points by simonpure 1y ago

bryan0 1y ago
The Matryoshka embeddings seem interesting: > The Gemini embedding model, gemini-embedding-001, is trained using the Matryoshka Representation Learning (MRL) technique which teaches a model to learn high-dimensional embeddings that have initial segments (or prefixes) which are also useful, simpler versions of the same data. Use the output_dimensionality parameter to control the size of the output embedding vector. Selecting a smaller output dimensionality can save storage space and increase computational efficiency for downstream applications, while sacrificing little in terms of quality. By default, it outputs a 3072-dimensional embedding, but you can truncate it to a smaller size without losing quality to save storage space. We recommend using 768, 1536, or 3072 output dimensions. [0] looks like even the 256-dim embeddings perform really well. [0]: https://ai.google.dev/gemini-api/docs/embeddings#quality-for-smaller-dimensions https://ai.google.dev/gemini-api/docs/embeddings#quality-for...
- simonw 1y ago
  The Matryoshka trick is really neat - there's a good explanation here: https://huggingface.co/blog/matryoshka https://huggingface.co/blog/matryoshka I've seen it in a few models now - Nomic Embed 1.5 was the first https://www.nomic.ai/blog/posts/nomic-embed-matryoshka https://www.nomic.ai/blog/posts/nomic-embed-matryoshka
  - alach11 1y ago
    OpenAI did it a few weeks earlier when they released text-embedding-3-large, right?
    - simonw 1y ago
      Huh, yeah you're right: that was January 25th 2024 https://openai.com/index/new-embedding-models-and-api-updates/ https://openai.com/index/new-embedding-models-and-api-update... Nomic 1.5 was February 14th 2024: https://www.nomic.ai/blog/posts/nomic-embed-matryoshka https://www.nomic.ai/blog/posts/nomic-embed-matryoshka
- ACCount36 1y ago
  Google teams seem to be in love with that Matryoshka tech. I wonder how far that scales.
  - OutOfHere 1y ago
    It's a practical feature. Scaling is irrelevant in this context because it scales to the length of the embedding, although in batches of k-length embeddings.
- thefourthchime 1y ago
  It's interesting, but the improvement they're claiming isn't that groundbreaking.
- OutOfHere 1y ago
  Does OpenAI's text-embedding-3-large or text-embedding-3-small embedding model have the Matryoshka property?
  - minimaxir 1y ago
    They do, they just don't advertise it well (and only confirmed it with a footnote after criticism of its omission): https://openai.com/index/new-embedding-models-and-api-updates/ https://openai.com/index/new-embedding-models-and-api-update... > Both of our new embedding models were trained with a technique that allows developers to trade-off performance and cost of using embeddings. Specifically, developers can shorten embeddings (i.e. remove some numbers from the end of the sequence) without the embedding losing its concept-representing properties by passing in the dimensions API parameter. For example, on the MTEB benchmark, a text-embedding-3-large embedding can be shortened to a size of 256 while still outperforming an unshortened text-embedding-ada-002 embedding with a size of 1536.
mvieira38 1y ago
To anyone working in these types of applications, are embeddings still worth it compared to agentic search for text? If I have a directory of text files, for example, is it better to save all of their embeddings in a VDB and use that, or are LLMs now good enough that I can just let them use ripgrep or something to search for themselves?
- philip1209 1y ago
  Semantic search is still important. I'd say that regex search is also quickly rising in importance, especially for coding agents.
- pjm331 1y ago
  With the caveat that I have not spent a serious amount of time trying to get RAG to work - my brief attempt to use it via AWS knowledge base to compare it vs agentic search resulted in me sticking with agentic search (via Claude code SDK) My impression was there’s lots of knobs you can tune with RAG and it’s just more complex in general - so maybe there’s a point where the amount of text I have is large enough that that complexity pays off - but right now agentic search works very well and is significantly simpler to get started with
- simonw 1y ago
  If your LLM is good enough you'll likely get better results from tool calling with grep or a FTS engine - the better models can even adapt their search patterns to search for things like "dog OR canine" where previously vector similarity may have been a bigger win. Getting embeddings working takes a bunch of work: you need to decide on a chunking strategy, then run the embeddings, then decide how best to store them for fast retrieval. You often end up having to keep your embedding store in memory which can add up for larger volumes of data. I did a whole lot of work with embeddings last year but I've mostly lost interest now that tool-based-search has become so powerful. Hooking up tool-based-search that itself uses embeddings is worth exploring, but you may find that the results you get from ripgrep are good enough that it's not worth the considerable extra effort.
- deleted 1y ago
  [deleted]
morkalork 1y ago
Question to other GCP users, how are you finding Google's aggressive deprecation of older embedding models? Feels like you have to pay to rerun your data through every 12 months.
- adregan 1y ago
  This is precisely the risk I’ve been wondering about with vectorization. I’ve considered that an open source model might be valuable as you could always find someone to host it for you and control the deprecation rate yourself.
- throwaway-blaze 1y ago
  You know of lots of LLM-using apps that don't need to re-run their fine tunings or embeddings because of improvements or new features at least annually? Things are moving so fast that "every 12 months" seems kinda slow...
- BoorishBears 1y ago
  My costs for embedding are so small compared to inference I don't generally notice. But am I crazy or did the pre-production version of gemini-embedding-001 have a much larger max context length? Edit: It seems like it did? 8k -> 2k? Huge downgrade if true, I was really excited about the experimental model reaching GA before that
asdev 1y ago
I feel like tool calling killed RAG, however you have less control over how the retrieved data is injected in the context.
- OutOfHere 1y ago
  How would you use tool-calling to filter through millions of documents? You need some search functionality, whether old-school search or embedding search. If you have only thousands of documents, then sure, you don't need search, as you can feed them all to the LLM.
  - kfajdsl 1y ago
    You give the LLM search tools.
    - OutOfHere 1y ago
      That's missing the point. You are hiding the search behind the tool, but it's still search. Whether you use a tool or a hardcoded workflow is irrelevant.
  - kridsdale1 1y ago
    I haven’t built either system but it seems clear that tool calling will be ‘O(num_targets * O(search tool))’, while RAG will be ‘O(embed_query * num_targets)’. RAG looks linear (constant per lookup) while tools look polynomial. And tools will possibly fill up the limited LLM context too.
- deleted 1y ago
  [deleted]
- billmalarky 1y ago
  Search tool calling is RAG. Maybe we should call it a "RAG Agent" to be more en vogue heh. But RAG is not just similarity search on embeddings in vector DBs. RAG is any type of a retrieval + context injection step prior to inference. Heck, the RAG Agent could run cosign diff on your vector db in addition to grep, FTS queries, KB api calls, whatever, to do wide recall (candidate generation) then rerank (relevance prioritization) all the results. You are probably correct that for most use cases search tool calling makes more practical sense than embeddings similarity search to power RAG.
stillpointlab 1y ago
> Embeddings are crucial here, as they efficiently identify and integrate vital information—like documents, conversation history, and tool definitions—directly into a model's working memory. I feel like I'm falling behind here, but can someone explain this to me? My high-level view of embedding is that I send some text to the provider, they tokenize the text and then run it through some NN that spits out a vector of numbers of a particular size (looks to be variable in this case including 768, 1536 and 3072). I can then use those embeddings in places like a vector DB where I might want to do some kind of similarity search (e.g. cosine difference). I can also use them to do clustering on that similarity which can give me some classification capabilities. But how does this translate to these things being "directly into a model's working memory'? My understanding is that with RAG I just throw a bunch of the embeddings into a vector DB as keys but the ultimate text I send in the context to the LLM is the source text that the keys represent. I don't actually send the embeddings themselves to the LLM. So what is is marketing stuff about "directly into a model's working memory."? Is my mental view wrong?
- NicholasD43 1y ago
  You're right on this. "Advanced" RAG techniques are all complete marketing BS, in the end all you're doing it passing the text into the model's context window.
- letitgo12345 1y ago
  LLMs can use search engines as a tool. One possibility is Google embeds the search query through these embeddings and does retrieval using them and then the retrieved result is pasted into the model's chain of thought (which..unless they have an external memory module in their model, is basically the model's only working memory).
  - stillpointlab 1y ago
    I'm reading the docs and it does not appear Google keeps these embeddings at all. I send some text to them, they return the embedding for that text at the size I specified. So the flow is something like: 1. Have a text doc (or library of docs) 2. Chunk it into small pieces 3. Send each chunk to <provider> and get an embedding vector of some size back 4. Use the embedding to: 4a. Semantic search / RAG: put the embeddings in a vector DB and do some similarity search on the embedding. The ultimate output is the source chunk 4b. Run a cluster algorithm on the embedding to generate some kind of graph representation of my data 4c. Run a classifier algorithm on the embedding to allow me to classify new data 5. The output of all steps in 4 is crucially text 6. Send that text to an LLM At no point is the embedding directly in the models memory.
deleted 1y ago
[deleted]
mijoharas 1y ago
What open embeddings models would people recommend. Still Nomic?
- christina97 1y ago
  The Qwen3 embedding models were released recently and do very well on benchmarks.
- hereme888 1y ago
  I'm using the qwen3 4B model in consumer hardware, which beats Gemini in English language tasks.
- gnulinux 1y ago
  Qwen3 is the open weight state of the art at the moment. Qwen3-embedding-8B and Qwen3-reranker-8b are surprisingly good (according to some benchmarks, better than Gemini 2.5 embedding). 4B is also nearly as good so you might as well use that too unless 8B benefits your usecase. If you don't need a SOTA-precise embedding model because you'll run a more powerful reranker, you could run qwen3-embedding-4B at Q4 which is only 2GB, and will process extremely fast in most hardware. A weaker but close choice is `Qwen3-Embedding-0.6B` at Q8 which is about 600MB and will run just fine on most powerful CPUs. So if that does the job for you, you may not even need GPU, just grab an instance with 16 vCPUs, that'll give you plenty of throughput, probably more than you need until your RAG has thousands of active users.
dmezzetti 1y ago
It's always worth checking out the MTEB leaderboard: https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard There are some good open models there that have longer context limits and fewer dimensions. The benchmarks are just a guide. It's best to build a test dataset with your own data. This is a good example of that: https://github.com/beir-cellar/beir/wiki/Load-your-custom-dataset https://github.com/beir-cellar/beir/wiki/Load-your-custom-da... Another benefit of having your own test dataset, is that it can grow as your data grows. And you can quickly test new models to see how it performs with YOUR data.
miohtama 1y ago
> Everlaw, a platform providing verifiable RAG to help legal professionals analyze large volumes of discovery documents, requires precise semantic matching across millions of specialized texts. Through internal benchmarks, Everlaw found gemini-embedding-001 to be the best, achieving 87% accuracy in surfacing relevant answers from 1.4 million documents filled with industry-specific and complex legal terms, surpassing Voyage (84%) and OpenAI (73%) models. Furthermore, Gemini Embedding's Matryoshka property enables Everlaw to use compact representations, focusing essential information in fewer dimensions. This leads to minimal performance loss, reduced storage costs, and more efficient retrieval and search. This will make a lot of junior lawyers or their work obsolete. Here is a good podcast on the topic how will AI affect legal industry https://open.spotify.com/episode/4IAHG68BeGZzr9uHXYvu5z?si=qsh7K4xcS7y5aLedmRczVw https://open.spotify.com/episode/4IAHG68BeGZzr9uHXYvu5z?si=q...
- dlojudice 1y ago
  It's really cool to see Odd Lots being mentioned here on HN. It's one of my favorite podcasts. However, I think the guest for this particular episode wasn't up to the task of answering questions and exploring the possibilities of using AI in the legal world.
jcims 1y ago
I'm short on vocabulary here but it seems that using content embedding similarity to find relevant (chunks of) content to feed an LLM is orthogonal to the use of LLMs to take automatically curated content chunks and use them to enrich a context. Is that correct? I'm just curious why this type of content selection seems to have been popularized and in many ways become the defacto standard for RAG, and (as far as I know but I haven't looked at 'search' in a long time) not generally used for general purpose search?
- elliotto 1y ago
  What do you mean by automatically curated content chunks? RAG with Embedding search is the process of deciding which chunks go into the context of the bot so that it can reference them to answer a user question
  - jcims 1y ago
    I guess I'm saying that over the past 30 years there have been a number of systems developed that take input from a user and find relevant bits of content from some corpus...aka 'search'. Searches using vector embeddings are likely better at matching relevant semantics than most other systems, so they are an excellent candidate for RAG. However, if there's a system that's already working quite well at finding relevant content based on user input, then there wouldn't necessarily be any value in adding a vectorized search to the RAG pipeline. Just use the existing system to populate relevant content into the context. Then the other half of my wondering is why the primary use case for vector databases appears (?) to be for RAG and not just a general purpose search engine.
    - elliotto 1y ago
      Ah I understand. My startup provides a vector search system as part of its offering. A user can upload a dataset of records and build a vector index on one of its columns and perform searches. It honestly works incredibly well on a whole bunch of different domains and I was shocked at how useful it was out of the box compared to a conventional BM25 style keyword search. Since we've got this working it's completely changed the way I think about navigating unstructured text data. If I have a dataset of 100k company website scrapes and I was looking for gyms, and I did a search for 'gym' I would get a whole bunch of conventional gyms. But I would miss companies that described themselves at fitness centers, or aquatic centers or MMA dojos. Vector search picks all of these up, but usually ranks them slightly lower. If I'm building a RAG bot that is helping me look up companies and I search for a gym, I want the bot to have these extra companies in its context. I can do a vector similarity cutoff, but I can also do a #records cutoff, so that it always has the X most relevant records in its context window. We've found the fuzziness of the vector search to be a problem in general purpose search cases because people write searches optimizing for keyword match. We had this problem with a company using it for a dataset with highly technical product codes that the embedding search was missing. Our solution was a hybrid keyword / vector search system for these guys that prioritized keyword match but also considered vector similarity. But it's still a big issue to communicate to the user what to write in the embedding search box - whereas in RAG the bot handles all of this. I think it's an unsolved problem and there continuous to be enormous development in this space.
djoldman 1y ago
It may be worth pointing out that a few open weights models score higher than gemini-embedding-001 on MTEB: https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard Particularly Qwen3-Embedding-8B and Qwen3-Embedding-4B: https://huggingface.co/Qwen/Qwen3-Embedding-8B https://huggingface.co/Qwen/Qwen3-Embedding-8B
- electroglyph 1y ago
  i don't think many people are having luck replicating those benchmarks, the models are a bit weird
  - asaddhamani 1y ago
    I can't trust MTEB as there's been a huge difference between benchmark scores and actual performance. I made a small tool to help me compare various embedding models: https://www.vectorsimilaritytest.com/ https://www.vectorsimilaritytest.com/ Qwen embedding models score very highly but are highly sensitive to word order (they use last token pooling which simplified means they only look at the last word of the input). Change the word order and the scores change completely. Voyage models score highly too, but changing a word from singular to plural can again completely change the scores. I find myself doing a hybrid search, rerank and shortlist the results, then feed them to an LLM to judge what is and isn't relevant.
keizo 1y ago
has anyone done some simple latency profiling of gemini embedding vs open ai embedding api? seem like that api call is one of the biggest chunks of time in a simple rag setup.
- elliotto 1y ago
  In my experience the api call is trivial compared to the time taken for the LLM to compose the response.
  - keizo 1y ago
    gemini flash and groq are pretty fast, and that part is streamable. curiosity got the best of me so i had claude code write a quick test. given this test is simply is 20 requests, with 1 second delay between requests ran once. so take with a grain of salt, but interesting still. Extra half second in a search is super noticeable so google looking like a reasonable improvement. OpenAI Statistics: - Average: 0.360 seconds - Median: 0.292 seconds - Min: 0.211 seconds - Max: 0.779 seconds - Std Dev: 0.172 seconds Google Gemini Statistics: - Average: 0.304 seconds - Median: 0.273 seconds - Min: 0.250 seconds - Max: 0.445 seconds - Std Dev: 0.066 seconds The key insights from these numbers: - Google has much lower standard deviation (0.066 vs 0.172), meaning more consistent/predictable performance - Google's worst-case (max) is much better than OpenAI's (0.445s vs 0.779s) - OpenAI had a slightly better best-case (min) performance (0.211s vs 0.250s) - Google's performance is more tightly clustered around its average, while OpenAI has more variability
aziis98 1y ago
I'm just can't wait for a globally scaled rag system. I think that will be a turning point for search engines. For now there is only https://exa.ai/ https://exa.ai/ that is currently doing something similar it seems.
zapnuk 1y ago
Good luck to anyone using it. We used it for embedding about 6k documents. The API constantly gives you quota errors when you reach about 150 requests/min eventhough the quota should allow about 50_000 requests/min. We’d like to use the Batch API, but the model isn’t available yet. Quite a nice model though. Being able to get embeddings for a specific task type [1] is very interesting. We used classification specific embeddings and noticed a meaningful improvment when we used the embeddings as input for a classifier. 1: https://ai.google.dev/gemini-api/docs/embeddings#supported-task-types https://ai.google.dev/gemini-api/docs/embeddings#supported-t...
- ofisboy 1y ago
  Same here. I tested gemini embeddings api for 1 to 5,000ish social media comments. It filled up the quota almost immediately. Since then, I’m just using qwen embeddings locally. Open source, free and relatively comparable.
curl-up 1y ago
Anyone who has recently worked on embedding model finetuning, any useful tools you'd recommend (both for dataset curation and actual finetuning)? Any models you'd recommend as especially good for finetuning? I'm interested in both full model finetunes, and downstream matrix optimization as done in [1]. [1] https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb https://github.com/openai/openai-cookbook/blob/main/examples...
nikolayasdf123 1y ago
no image support is a deal breaker. multi-modality is a must
nikolayasdf123 1y ago
interesting. high quality optimized embeddings is very nice to have
jgalt212 1y ago
Is one LLM embedding much better than another? To me, if you're building a vector database off embeddings, it's best and not punitive to stick to a self hosted public weights model.
TN1ck 1y ago
VP of Engineering of re:cap here (featured in the article), if anybody has any more detailed questions, happy to answer!
_Chief 1y ago
I have been thinking around solving this problem. I think one of the reasons some AI assistants shine vs others is how they can reduce the amount of context the LLM needs to work with using in-built tools. I think there's room to democratize these capabilities. One such capability is allowing the LLMs to directly work with the embeddings. I wrote an MCP server directory-indexer[1] for this (self-hosted indexing mcp server). The goal being indexing any directories you want your AI to know about and gives the it MCP tools to search through the embeddings etc. While an agentic grep may be valuable, when working with tons of files with similar topics (like customer cases, technical docs), pre-processed embeddings have proven valuable for me. One reason I really like it is that it democratizes my data and documents: giving consistent results when working with different AI assistants - the alternative being vastly different results based on the in-built capabilities of the coding assistants. Another being having access to you "knowledge" from any project you're on. Though since this is selfhosted, I use nomic-embed-text for the embedding which has been sufficient for most use cases. [1] https://github.com/peteretelej/directory-indexer https://github.com/peteretelej/directory-indexer
- FeepingCreature 1y ago
  Shit man, I want a commandline tool that can grep embeddings. `emb-locate` when

278 points by simonpure 1y ago

The Matryoshka embeddings seem interesting: > The Gemini embedding model, gemini-embedding-001, is trained using the Matryoshka Representation Learning (MRL) technique which teaches a model to learn high-dimensional embeddings that have initial segments (or prefixes) which are also useful, simpler versions of the same data. Use the output_dimensionality parameter to control the size of the output embedding vector. Selecting a smaller output dimensionality can save storage space and increase computational efficiency for downstream applications, while sacrificing little in terms of quality. By default, it outputs a 3072-dimensional embedding, but you can truncate it to a smaller size without losing quality to save storage space. We recommend using 768, 1536, or 3072 output dimensions. [0] looks like even the 256-dim embeddings perform really well. [0]: https://ai.google.dev/gemini-api/docs/embeddings#quality-for-smaller-dimensions https://ai.google.dev/gemini-api/docs/embeddings#quality-for...

simonw 1y ago
The Matryoshka trick is really neat - there's a good explanation here: https://huggingface.co/blog/matryoshka https://huggingface.co/blog/matryoshka I've seen it in a few models now - Nomic Embed 1.5 was the first https://www.nomic.ai/blog/posts/nomic-embed-matryoshka https://www.nomic.ai/blog/posts/nomic-embed-matryoshka
- alach11 1y ago
  OpenAI did it a few weeks earlier when they released text-embedding-3-large, right?
  - simonw 1y ago
    Huh, yeah you're right: that was January 25th 2024 https://openai.com/index/new-embedding-models-and-api-updates/ https://openai.com/index/new-embedding-models-and-api-update... Nomic 1.5 was February 14th 2024: https://www.nomic.ai/blog/posts/nomic-embed-matryoshka https://www.nomic.ai/blog/posts/nomic-embed-matryoshka
ACCount36 1y ago
Google teams seem to be in love with that Matryoshka tech. I wonder how far that scales.
- OutOfHere 1y ago
  It's a practical feature. Scaling is irrelevant in this context because it scales to the length of the embedding, although in batches of k-length embeddings.
thefourthchime 1y ago
It's interesting, but the improvement they're claiming isn't that groundbreaking.
OutOfHere 1y ago
Does OpenAI's text-embedding-3-large or text-embedding-3-small embedding model have the Matryoshka property?
- minimaxir 1y ago
  They do, they just don't advertise it well (and only confirmed it with a footnote after criticism of its omission): https://openai.com/index/new-embedding-models-and-api-updates/ https://openai.com/index/new-embedding-models-and-api-update... > Both of our new embedding models were trained with a technique that allows developers to trade-off performance and cost of using embeddings. Specifically, developers can shorten embeddings (i.e. remove some numbers from the end of the sequence) without the embedding losing its concept-representing properties by passing in the dimensions API parameter. For example, on the MTEB benchmark, a text-embedding-3-large embedding can be shortened to a size of 256 while still outperforming an unshortened text-embedding-ada-002 embedding with a size of 1536.

mvieira38 1y ago

To anyone working in these types of applications, are embeddings still worth it compared to agentic search for text? If I have a directory of text files, for example, is it better to save all of their embeddings in a VDB and use that, or are LLMs now good enough that I can just let them use ripgrep or something to search for themselves?

philip1209 1y ago
Semantic search is still important. I'd say that regex search is also quickly rising in importance, especially for coding agents.
pjm331 1y ago
With the caveat that I have not spent a serious amount of time trying to get RAG to work - my brief attempt to use it via AWS knowledge base to compare it vs agentic search resulted in me sticking with agentic search (via Claude code SDK) My impression was there’s lots of knobs you can tune with RAG and it’s just more complex in general - so maybe there’s a point where the amount of text I have is large enough that that complexity pays off - but right now agentic search works very well and is significantly simpler to get started with
simonw 1y ago
If your LLM is good enough you'll likely get better results from tool calling with grep or a FTS engine - the better models can even adapt their search patterns to search for things like "dog OR canine" where previously vector similarity may have been a bigger win. Getting embeddings working takes a bunch of work: you need to decide on a chunking strategy, then run the embeddings, then decide how best to store them for fast retrieval. You often end up having to keep your embedding store in memory which can add up for larger volumes of data. I did a whole lot of work with embeddings last year but I've mostly lost interest now that tool-based-search has become so powerful. Hooking up tool-based-search that itself uses embeddings is worth exploring, but you may find that the results you get from ripgrep are good enough that it's not worth the considerable extra effort.
deleted 1y ago
[deleted]

morkalork 1y ago

Question to other GCP users, how are you finding Google's aggressive deprecation of older embedding models? Feels like you have to pay to rerun your data through every 12 months.

adregan 1y ago
This is precisely the risk I’ve been wondering about with vectorization. I’ve considered that an open source model might be valuable as you could always find someone to host it for you and control the deprecation rate yourself.
throwaway-blaze 1y ago
You know of lots of LLM-using apps that don't need to re-run their fine tunings or embeddings because of improvements or new features at least annually? Things are moving so fast that "every 12 months" seems kinda slow...
BoorishBears 1y ago
My costs for embedding are so small compared to inference I don't generally notice. But am I crazy or did the pre-production version of gemini-embedding-001 have a much larger max context length? Edit: It seems like it did? 8k -> 2k? Huge downgrade if true, I was really excited about the experimental model reaching GA before that

asdev 1y ago

I feel like tool calling killed RAG, however you have less control over how the retrieved data is injected in the context.

OutOfHere 1y ago
How would you use tool-calling to filter through millions of documents? You need some search functionality, whether old-school search or embedding search. If you have only thousands of documents, then sure, you don't need search, as you can feed them all to the LLM.
- kfajdsl 1y ago
  You give the LLM search tools.
  - OutOfHere 1y ago
    That's missing the point. You are hiding the search behind the tool, but it's still search. Whether you use a tool or a hardcoded workflow is irrelevant.
- kridsdale1 1y ago
  I haven’t built either system but it seems clear that tool calling will be ‘O(num_targets * O(search tool))’, while RAG will be ‘O(embed_query * num_targets)’. RAG looks linear (constant per lookup) while tools look polynomial. And tools will possibly fill up the limited LLM context too.
deleted 1y ago
[deleted]
billmalarky 1y ago
Search tool calling is RAG. Maybe we should call it a "RAG Agent" to be more en vogue heh. But RAG is not just similarity search on embeddings in vector DBs. RAG is any type of a retrieval + context injection step prior to inference. Heck, the RAG Agent could run cosign diff on your vector db in addition to grep, FTS queries, KB api calls, whatever, to do wide recall (candidate generation) then rerank (relevance prioritization) all the results. You are probably correct that for most use cases search tool calling makes more practical sense than embeddings similarity search to power RAG.

stillpointlab 1y ago

> Embeddings are crucial here, as they efficiently identify and integrate vital information—like documents, conversation history, and tool definitions—directly into a model's working memory. I feel like I'm falling behind here, but can someone explain this to me? My high-level view of embedding is that I send some text to the provider, they tokenize the text and then run it through some NN that spits out a vector of numbers of a particular size (looks to be variable in this case including 768, 1536 and 3072). I can then use those embeddings in places like a vector DB where I might want to do some kind of similarity search (e.g. cosine difference). I can also use them to do clustering on that similarity which can give me some classification capabilities. But how does this translate to these things being "directly into a model's working memory'? My understanding is that with RAG I just throw a bunch of the embeddings into a vector DB as keys but the ultimate text I send in the context to the LLM is the source text that the keys represent. I don't actually send the embeddings themselves to the LLM. So what is is marketing stuff about "directly into a model's working memory."? Is my mental view wrong?

NicholasD43 1y ago
You're right on this. "Advanced" RAG techniques are all complete marketing BS, in the end all you're doing it passing the text into the model's context window.
letitgo12345 1y ago
LLMs can use search engines as a tool. One possibility is Google embeds the search query through these embeddings and does retrieval using them and then the retrieved result is pasted into the model's chain of thought (which..unless they have an external memory module in their model, is basically the model's only working memory).
- stillpointlab 1y ago
  I'm reading the docs and it does not appear Google keeps these embeddings at all. I send some text to them, they return the embedding for that text at the size I specified. So the flow is something like: 1. Have a text doc (or library of docs) 2. Chunk it into small pieces 3. Send each chunk to <provider> and get an embedding vector of some size back 4. Use the embedding to: 4a. Semantic search / RAG: put the embeddings in a vector DB and do some similarity search on the embedding. The ultimate output is the source chunk 4b. Run a cluster algorithm on the embedding to generate some kind of graph representation of my data 4c. Run a classifier algorithm on the embedding to allow me to classify new data 5. The output of all steps in 4 is crucially text 6. Send that text to an LLM At no point is the embedding directly in the models memory.

deleted 1y ago

[deleted]

mijoharas 1y ago

What open embeddings models would people recommend. Still Nomic?

christina97 1y ago
The Qwen3 embedding models were released recently and do very well on benchmarks.
hereme888 1y ago
I'm using the qwen3 4B model in consumer hardware, which beats Gemini in English language tasks.
gnulinux 1y ago
Qwen3 is the open weight state of the art at the moment. Qwen3-embedding-8B and Qwen3-reranker-8b are surprisingly good (according to some benchmarks, better than Gemini 2.5 embedding). 4B is also nearly as good so you might as well use that too unless 8B benefits your usecase. If you don't need a SOTA-precise embedding model because you'll run a more powerful reranker, you could run qwen3-embedding-4B at Q4 which is only 2GB, and will process extremely fast in most hardware. A weaker but close choice is `Qwen3-Embedding-0.6B` at Q8 which is about 600MB and will run just fine on most powerful CPUs. So if that does the job for you, you may not even need GPU, just grab an instance with 16 vCPUs, that'll give you plenty of throughput, probably more than you need until your RAG has thousands of active users.

dmezzetti 1y ago

It's always worth checking out the MTEB leaderboard: https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard There are some good open models there that have longer context limits and fewer dimensions. The benchmarks are just a guide. It's best to build a test dataset with your own data. This is a good example of that: https://github.com/beir-cellar/beir/wiki/Load-your-custom-dataset https://github.com/beir-cellar/beir/wiki/Load-your-custom-da... Another benefit of having your own test dataset, is that it can grow as your data grows. And you can quickly test new models to see how it performs with YOUR data.

miohtama 1y ago

> Everlaw, a platform providing verifiable RAG to help legal professionals analyze large volumes of discovery documents, requires precise semantic matching across millions of specialized texts. Through internal benchmarks, Everlaw found gemini-embedding-001 to be the best, achieving 87% accuracy in surfacing relevant answers from 1.4 million documents filled with industry-specific and complex legal terms, surpassing Voyage (84%) and OpenAI (73%) models. Furthermore, Gemini Embedding's Matryoshka property enables Everlaw to use compact representations, focusing essential information in fewer dimensions. This leads to minimal performance loss, reduced storage costs, and more efficient retrieval and search. This will make a lot of junior lawyers or their work obsolete. Here is a good podcast on the topic how will AI affect legal industry https://open.spotify.com/episode/4IAHG68BeGZzr9uHXYvu5z?si=qsh7K4xcS7y5aLedmRczVw https://open.spotify.com/episode/4IAHG68BeGZzr9uHXYvu5z?si=q...

dlojudice 1y ago
It's really cool to see Odd Lots being mentioned here on HN. It's one of my favorite podcasts. However, I think the guest for this particular episode wasn't up to the task of answering questions and exploring the possibilities of using AI in the legal world.

jcims 1y ago

I'm short on vocabulary here but it seems that using content embedding similarity to find relevant (chunks of) content to feed an LLM is orthogonal to the use of LLMs to take automatically curated content chunks and use them to enrich a context. Is that correct? I'm just curious why this type of content selection seems to have been popularized and in many ways become the defacto standard for RAG, and (as far as I know but I haven't looked at 'search' in a long time) not generally used for general purpose search?

elliotto 1y ago
What do you mean by automatically curated content chunks? RAG with Embedding search is the process of deciding which chunks go into the context of the bot so that it can reference them to answer a user question
- jcims 1y ago
  I guess I'm saying that over the past 30 years there have been a number of systems developed that take input from a user and find relevant bits of content from some corpus...aka 'search'. Searches using vector embeddings are likely better at matching relevant semantics than most other systems, so they are an excellent candidate for RAG. However, if there's a system that's already working quite well at finding relevant content based on user input, then there wouldn't necessarily be any value in adding a vectorized search to the RAG pipeline. Just use the existing system to populate relevant content into the context. Then the other half of my wondering is why the primary use case for vector databases appears (?) to be for RAG and not just a general purpose search engine.
  - elliotto 1y ago
    Ah I understand. My startup provides a vector search system as part of its offering. A user can upload a dataset of records and build a vector index on one of its columns and perform searches. It honestly works incredibly well on a whole bunch of different domains and I was shocked at how useful it was out of the box compared to a conventional BM25 style keyword search. Since we've got this working it's completely changed the way I think about navigating unstructured text data. If I have a dataset of 100k company website scrapes and I was looking for gyms, and I did a search for 'gym' I would get a whole bunch of conventional gyms. But I would miss companies that described themselves at fitness centers, or aquatic centers or MMA dojos. Vector search picks all of these up, but usually ranks them slightly lower. If I'm building a RAG bot that is helping me look up companies and I search for a gym, I want the bot to have these extra companies in its context. I can do a vector similarity cutoff, but I can also do a #records cutoff, so that it always has the X most relevant records in its context window. We've found the fuzziness of the vector search to be a problem in general purpose search cases because people write searches optimizing for keyword match. We had this problem with a company using it for a dataset with highly technical product codes that the embedding search was missing. Our solution was a hybrid keyword / vector search system for these guys that prioritized keyword match but also considered vector similarity. But it's still a big issue to communicate to the user what to write in the embedding search box - whereas in RAG the bot handles all of this. I think it's an unsolved problem and there continuous to be enormous development in this space.

djoldman 1y ago

It may be worth pointing out that a few open weights models score higher than gemini-embedding-001 on MTEB: https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard Particularly Qwen3-Embedding-8B and Qwen3-Embedding-4B: https://huggingface.co/Qwen/Qwen3-Embedding-8B https://huggingface.co/Qwen/Qwen3-Embedding-8B

electroglyph 1y ago
i don't think many people are having luck replicating those benchmarks, the models are a bit weird
- asaddhamani 1y ago
  I can't trust MTEB as there's been a huge difference between benchmark scores and actual performance. I made a small tool to help me compare various embedding models: https://www.vectorsimilaritytest.com/ https://www.vectorsimilaritytest.com/ Qwen embedding models score very highly but are highly sensitive to word order (they use last token pooling which simplified means they only look at the last word of the input). Change the word order and the scores change completely. Voyage models score highly too, but changing a word from singular to plural can again completely change the scores. I find myself doing a hybrid search, rerank and shortlist the results, then feed them to an LLM to judge what is and isn't relevant.

keizo 1y ago

has anyone done some simple latency profiling of gemini embedding vs open ai embedding api? seem like that api call is one of the biggest chunks of time in a simple rag setup.

elliotto 1y ago
In my experience the api call is trivial compared to the time taken for the LLM to compose the response.
- keizo 1y ago
  gemini flash and groq are pretty fast, and that part is streamable. curiosity got the best of me so i had claude code write a quick test. given this test is simply is 20 requests, with 1 second delay between requests ran once. so take with a grain of salt, but interesting still. Extra half second in a search is super noticeable so google looking like a reasonable improvement. OpenAI Statistics: - Average: 0.360 seconds - Median: 0.292 seconds - Min: 0.211 seconds - Max: 0.779 seconds - Std Dev: 0.172 seconds Google Gemini Statistics: - Average: 0.304 seconds - Median: 0.273 seconds - Min: 0.250 seconds - Max: 0.445 seconds - Std Dev: 0.066 seconds The key insights from these numbers: - Google has much lower standard deviation (0.066 vs 0.172), meaning more consistent/predictable performance - Google's worst-case (max) is much better than OpenAI's (0.445s vs 0.779s) - OpenAI had a slightly better best-case (min) performance (0.211s vs 0.250s) - Google's performance is more tightly clustered around its average, while OpenAI has more variability

aziis98 1y ago

I'm just can't wait for a globally scaled rag system. I think that will be a turning point for search engines. For now there is only https://exa.ai/ https://exa.ai/ that is currently doing something similar it seems.

zapnuk 1y ago

Good luck to anyone using it. We used it for embedding about 6k documents. The API constantly gives you quota errors when you reach about 150 requests/min eventhough the quota should allow about 50_000 requests/min. We’d like to use the Batch API, but the model isn’t available yet. Quite a nice model though. Being able to get embeddings for a specific task type [1] is very interesting. We used classification specific embeddings and noticed a meaningful improvment when we used the embeddings as input for a classifier. 1: https://ai.google.dev/gemini-api/docs/embeddings#supported-task-types https://ai.google.dev/gemini-api/docs/embeddings#supported-t...

ofisboy 1y ago
Same here. I tested gemini embeddings api for 1 to 5,000ish social media comments. It filled up the quota almost immediately. Since then, I’m just using qwen embeddings locally. Open source, free and relatively comparable.

curl-up 1y ago

Anyone who has recently worked on embedding model finetuning, any useful tools you'd recommend (both for dataset curation and actual finetuning)? Any models you'd recommend as especially good for finetuning? I'm interested in both full model finetunes, and downstream matrix optimization as done in [1]. [1] https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb https://github.com/openai/openai-cookbook/blob/main/examples...

nikolayasdf123 1y ago

no image support is a deal breaker. multi-modality is a must

nikolayasdf123 1y ago

interesting. high quality optimized embeddings is very nice to have

jgalt212 1y ago

Is one LLM embedding much better than another? To me, if you're building a vector database off embeddings, it's best and not punitive to stick to a self hosted public weights model.

TN1ck 1y ago

VP of Engineering of re:cap here (featured in the article), if anybody has any more detailed questions, happy to answer!

_Chief 1y ago

I have been thinking around solving this problem. I think one of the reasons some AI assistants shine vs others is how they can reduce the amount of context the LLM needs to work with using in-built tools. I think there's room to democratize these capabilities. One such capability is allowing the LLMs to directly work with the embeddings. I wrote an MCP server directory-indexer[1] for this (self-hosted indexing mcp server). The goal being indexing any directories you want your AI to know about and gives the it MCP tools to search through the embeddings etc. While an agentic grep may be valuable, when working with tons of files with similar topics (like customer cases, technical docs), pre-processed embeddings have proven valuable for me. One reason I really like it is that it democratizes my data and documents: giving consistent results when working with different AI assistants - the alternative being vastly different results based on the in-built capabilities of the coding assistants. Another being having access to you "knowledge" from any project you're on. Though since this is selfhosted, I use nomic-embed-text for the embedding which has been sufficient for most use cases. [1] https://github.com/peteretelej/directory-indexer https://github.com/peteretelej/directory-indexer

FeepingCreature 1y ago
Shit man, I want a commandline tool that can grep embeddings. `emb-locate` when