3 ms·
The retrieval part of RAG is usually vector search; but it doesn't have to be. Or at least not exclusively. I've worked with various search backends for about
by jillesvangurp 2y ago
The retrieval part of RAG is usually vector search; but it doesn't have to be. Or at least not exclusively.
I've worked with various search backends for about 20 years. People treat vector search like magic pixie dust but the reality is that it's not that great unless you heavily tune your models to your use cases. A well tuned manually crafted query goes a long way.
Pretty much any system I've built over the last few years, the best way to think about search is about building a search context that includes anything relevant to answering the user's question. The user's direct input is only a small part of that. In the case of mobile systems, the user entered query is actually typically a very minor part of it. People type two or three letters and then expect magic to happen. Vector search is completely useless in situations like that. Why does search on mobile work anyway? Because of everything else we know to create a query (user location, time zone, locale, past searches, preferences, etc.)
RAG isn't any different. It's just search where the search results are post processed by an LLM with whatever the user typed. The better the query and retrieval, the better the result. The LLM can't rescue a poorly tuned search. But it can dig through a massive result of search results and extract key points.
- cnity 2y agoThe valuable part _is_ the vector search, at least from the perspective of the OP. It sounds like what they're after is to just pass back the source texts from the vector search, rather than passing them back through an LLM to be summarised. I generally agree with the OP: in a search context I often don't want that last LLM summarisation step. I want embeddings generated from my search query for document lookup, and I want to see the original sources. Phind did (does?) something like this with citations, which is at least an improvement.
- danielbln 2y agoIt's common practice to have the LLM spit out the meta data as well, e.g. a source link. We do that for almost all of our usecases, it's the best way to increase trust in the response. The LLM step does provide a benefit, as it can synthesize a tight and concise answer to the user's question, instead of returning a list of source documents to sift through manually.
- wumbo 2y agoThe real beauty is not visiting people’s horrible web pages. Oh how I want information. Oh how I don’t want your newlsetter modal to obscure that information.
- PaulHoule 2y agoUpnext spent about two years developing a "reader" that could deenshittify most pages in the context of a "read-it-later" application. They pivoted to start Brief which does AI recommendation and summarization and shut down Upnext. I think they are still using the back end of the reader to get a clean extract of text to feed into the AI. But it's very telling that users are supportive of Google's agenda of not sending you to people's actual web sites because web sites have gotten so bad. Europeans will deny it but I think those cookie banners broke the dam so that now it is normalized that you can pop up four different modals asking for an email address.
- wumbo 2y agoNot Europe’s fault there, just determined adversarial web design.
- jmount 2y agoI forgot this point, but you are right. I use ChatGPT for all programming queries- as I don't want Google routing me to pop-up infested splogs.
- valicord 2y agoThen just use a regular search engine? RAG=search+LLM, if you don't want the LLM part, use Google, or whatever your favorite one is. Vector search by itself is not a novel concept, all major search engines already rely on it anyway.
- cnity 2y agoRight. The OP title is: "I want flexible queries, not RAG". I didn't write the article, btw!
- autokad 2y agoI think the op raises a genuine criticism of search. its brittle. I personally hate it, I know how to use google very well, it doesn't mean I like using it. Defenders say: but I want to be able to super fine tune my search with 50 flags and various combinations of quotes. That's just horrible. It should understand what you want by what you type. Sometimes, I don't even know the word I am trying to search, llm's do really well at that. Most of my search now happens in llm. Usually I want information, not a webpage. I do understand the writer's gripes with search, and it should have been fixed a long time ago.
- kohlerm 2y agoIt also very much depends on the quality of your documents. E.g. if you have a set of very good recipes, rather than a random set coming for the Internet, then you can get very good answers with RAG
- motoboi 2y ago> People treat vector search like magic pixie dust but the reality is that it's not that great unless you heavily tune your models to your use cases I believe that what most people miss is that embeddings are the product of the model. Embeddings have been getting so much better with the new models and so embedding-based search improved too. Embeddings are coordinates in a world representation that "organizes" information so that location in that world matters and serve as a way to differentiate meaning. In other worlds, word2vec was a simple and poor world representation. OpenAI embeddings are an astounding world representations coordinates.
- autokad 2y agoalso I think its about time. we don't have the time to super fine tune the document retrieval, and that magic pixie dust gets us 90-99% there. That extra 1% could take time management doesnt want to give.
- DeathArrow 2y ago>But it can dig through a massive result of search results and extract key points. And there is value in that.
- jillesvangurp 2y agoReducing the signal to noise ratio improves the results though. This is a classical search relevance problem.