6 ms·
I worked on large scale RAG systems before and can say people vastly underestimate full text search and vastly overestimate embeddings. FTS is really easy, port
by usernametaken29 2mo ago
I worked on large scale RAG systems before and can say people vastly underestimate full text search and vastly overestimate embeddings. FTS is really easy, portable and scalable and gets you very far, the 80/20 rule applies. Embeddings appear to be nice and magic but when you really get into them you notice: semantic similarity isn’t as good as you think and certainly it won’t make everyone happy. You will inevitably end up having to re-embed more or different chunks of your text to accommodate more and more precise embedding search - at which point you’ll go the last mile and do reranking etc etc all the while having to support the operational burden of vector search.
Then you turn around and build a search query with 500 keywords and sure it’s painful but it just works, accommodates all use cases, scales and is overall less annoying to maintain.
- clevergadget 2mo agoI don't know what level of quality is required for this site but RAG is trash its just trash. its magic beans.
- lacedeconstruct 2mo agoI thought text search was always the first thing you try, then fuzzy search, then you go for RAG
- ozim 2mo agoI think Bitwarden implemented some vector search in their password search feature ... totally annoying it gives me back all kinds of stuff that I don't care. I want fuzzy search like 95% of time and then I might consider having additional list of things that can be suggested by vector search.
- a1o 2mo agoA good UI could do these and also exact match, give some point system to the results, then order them and perhaps use a bold highlight to reflect what parts of the input query reflected in each result.
- gwerbin 2mo agoBandcamp has had legendarily bad semantic search for as long as they've been around. It's often completely impossible to find an artist or album or song even when you type the exact name.
- t_mahmood 2mo agoahh now I realize why I get so much completely irrelevant search results in many sites recently. I mean I'm searching for betel and you're giving me nuts. haha
- itintheory 2mo agoBitwarden has lost the plot. The most recent Windows update is so bad. It has way lower information density in the UI, more buttons to click for the same use, no longer puts focus on the search field by default (this one makes me irrationally angry), and on one of my Win 11 installs can't lock the vault, manually or automatically. How could they mess up such a simple app that worked fine for so long?! What perverse incentives caused this nonsense?!
- wongarsu 2mo agoIt's not like a simple embedding search takes that much longer to implement. Especially on short descriptions where you don't have to deal with chunking. And if you let an LLM write the code it's even less of a difference. Combine that with embedding search promising to solve all your search problems, and I understand why people often skip over full text search and go straight to embeddings
- j0selit0 2mo agoI wish everyone thought like you, in my experience unfortunately it's not the case
- EagnaIonat 2mo agoEven that is an oversimplification unless you are doing something very basic. Volume of documents, size of documents, versioning, frequency of update, documents similar or overlapping information, how much or exactly what you need for the LLM to understand, AI friendly documents, who has access and at what level, blue teaming, red teaming, multi-lingual, does the LLM know the domain language of the user and documents. I probably missed a few things even with that.
- kaon_2 2mo agoCan you elaborate? We have technicians searching in different languages. Also our knowledge base is often in different languages. I just don't see how full text search can work? Maybe in a problem space like a wiki where people always know what to search for?
- jon-wood 2mo agoInstinctively this feels like a two phase problem - start with some machine translation into a single spoken language and index that, then when people are querying do the same thing. When returning search results show them in the original language.
- whilenot-dev 2mo agoWhy not create indexes for multiple languages, as that would also avoid double translation issues (e.g. GER [query] → ENG [index] → GER [document])?
- j0selit0 2mo agoyou would also need to maintain multiple indexes in multiple languages. I never had to do that - but I assume it's a pain
- whilenot-dev 2mo agoIt's a matter of running a for-loop. You'd get faster response times (no query→index translation), but the storage requirements for the indexes would be larger.
- kaon_2 2mo agoYes we've tried. It works. But jargon is hard. RAG with embeddings works all the same. The LLM doesn't mind receiving sources in Italian, french and German, and then outputting the answer in Japanese while providing the verbatim German jargon term in brackets
- jameshart 2mo agoI think people also overestimate the need for full text search when the one doing the querying is an LLM. If your underlying data is structured records, like a customer database, while humans might not have time or skills to figure out that when they want to search by phone number they need to do a join from the contacts table to the users table and normalize the phone number to look up first, making it best to just surface phone numbers as part of the data that is full/text-indexed… an agent is quite happy to handcraft the right SQL to find records that match on a specific field, given the right SKILLS.md and schema information. Turning fuzzy searches into exact DB lookups is a great way LLMs can augment users. (Obviously this doesn’t apply to searching actual rich document data - for that, go all in on text search, embedding, etc)
- josh_p 2mo agoI worked on getting an address database into elasticsearch years ago when it was still using modified tf-idf. Customers wanted FTS where a lot of the queries would be something like "100 First Ave, NY" or "200 2nd St, MN". It was one of the most fun projects I've worked on in my career so far. I got a learn a lot about how US and international addresses worked, so many edge cases, and got to really understand how customers were using the existing search to make sure they weren't adding any duplicates to the database. Token filters and synonyms were neat and figuring out the right indexing strategy was a lot of fun. It was a lot more work to get it right for most of the use-cases our customers had than just "throw it into ES and be done". That would probably have been fine for the 80/20 case, like you said, but I agree that the bulk of the work is going to be fine-tuning the search solution, whatever technology you're using.
- oever 2mo agoWhat's your opinion on nominatim? I find that it gives up quickly when there's one or two typos in an address. It nails your examples.
- josh_p 2mo agoI didn't know this was a thing until you mentioned it. I'm glad there are more options today. I have since left the company and haven't gotten to work in address-related problems since. We were pretty invested in using Google at the time for their Geocoding API. The questions that needed answering for our use-case were: - Does this place actually exist? - Can it be found on Google Maps? - Do we (the company/researchers) have an entry for this place already?
- chorizo 2mo agoWhy not use what USPS Addresses API? That what it’s there for. It will validate and standardize addresses including ZIP+4. Wonder if other countries have similar API’s.
- polishdude20 2mo agoTheir API doesn't allow bulk conversions.
- mmargenot 2mo agoAnd you get bm25 for free with so many modern setups! I do still love to experiment with tuning semantic search for your specific corpus via various kinds of embeddings, but bm25 is hard to beat.
- quijoteuniv 2mo agoOn my last go at making my own rag i still got better results by collecting the data and uploading to a project in open(butclosed)ai. My own rag, used by an agent was giving poorer results, and even the agent prefered (derailed)to not use it and look for the info itself rather than using the rag
- idontneedcoffee 2mo agoI would be really grateful if someone could battle-test my frankendb in a full-fledged RAG setup(lmdb + roaring bitmaps + to-be-removed lance with a bitmap-based virtual fs-like tree on top of your data) outside of its original narrow use-case (index for user data + workflows) https://github.com/canvas-ui/canvas-synapsd https://github.com/canvas-ui/canvas-synapsd
- gardnr 2mo agoNice work.
- shay_ker 2mo agoHow long have "large scale RAG systems" really existed in the first place? I'm always surprised at this, given how new all this really is, relatively speaking.
- j0selit0 2mo agomy personal experience - I have been involved with such projects for the last 2 years. interestingly enough, a lot of times such initiatives didn't take off because people/stakeholders were overcomplicating things and wanting to use semantic search for everything - without having a minimum knowledge of chunking strategies, pros/cons etc
- bensyverson 2mo agoYes, and don’t forget, LLMs are very good at tagging, so it’s not even that painful to backfill the corpus.
- mdp2021 2mo ago> people vastly underestimate full text search It is not psychological, it is fully justified: substring search cannot find synonyms, periphrases and mistaken neighbours.
- locknitpicker 2mo ago> It is not psychological, it is fully justified: substring search cannot find synonyms, periphrases and mistaken neighbours. It is, if people don't even stop to think if they need synonyms, periphrases, or mistaken neighbours. As the blog post points out, more often than not you don't, particularly if your primary usecase is to search for technical keywords or codenames.
- lopsotronic 2mo agoPrecisely this. The people in charge of technical direction don't understand the fundamentals of the technology. So you get the idea that LLMs can help make sense of parts data. Which . . . no, no it really can't, not without ALSO plugging in basically every other hunk of natural language you might have laying around. Unless you think PLG HT HFI is just a natural synonym of HOT PLUG INJECTOR, in which case you're just quantitatively wrong. Vectors and LLMs are great, but there's no magic pill here. If your parts data and config management[1] is all crazy, that's an institutional problem. Buying a crapton of tokens isn't fixing it, unless you're using it to help build an actual formal solution based on good fundamentals. [1] Such as it is.
- dominotw 2mo ago> particularly if your primary usecase is to search for technical keywords or codenames. i dont believe ppl are building rag for this
- locknitpicker 2mo ago> i dont believe ppl are building rag for this What do you actually think people do when using LLMs to build AI coding agents?
- 2mo ago
- andai 2mo agoRe: the rube goldberg machine of diminishing returns https://www.anthropic.com/engineering/contextual-retrieval https://www.anthropic.com/engineering/contextual-retrieval This is from two years ago, but I think it's still SotA?
- gardnr 2mo agoThat is the approach I would take today. Late Interaction is worth a look. Evals are necessary.
- piterrro 2mo agoRAG only makes sense if you have an LLM review the results, pick the most relevant ones and iterate further if there's a need running another query and repeating the process. Raw dump of vector search (even with reranking) is asking for troubles (or rather weird user questions like 'why this crap popped up in the results?')
- _the_inflator 2mo agoRAG is art. I have a very straight forward setup that is highly modular. RAG is routing and decision making. I found so much joy in achieving the best results given the requirements than simply hoping for the best with the cool kid called vector db and embeddings. I agree with you. Depending on the context and required output I decide how to orchestrate a multitude of specialized modules that produce the best specific result to gain a universally usable system. It maintains itself. Also live updates need reruns and rebuilding certain indexes. Everything is highly dynamic but in a deterministic way. I found my niche with RAG selling and I build them myself. I take pride in them. So many look at the technology but not on the required output. It takes hours of talking to people to get an idea of what they need. And there are regulated businesses where certain information is required to be always factual correct - pricing for example. Vector search becomes a liability for this use case. So naturally you have to reconsider your system: mixing factual with probabilistic content and how to make sure, it hits always certain quality benchmarks and on the other hand doesn’t fail others. I love this kind of stuff. And there is personal information etc. Using modules is the key. Orchestration is really fun but I have to admit, not for the faint of heart. And ever changing parts: LLMs, or restrictions to be matched liked autonomously working - I love RAG. It gave me back the joy of developing. In fact I never had so much phun before, because it is also “team work”: I am not programming, I am managing a product. I was in Senior Management of a top tier international bank and besides that build the only ever working platform or IT transformation called dbCORE and overlooked 13 teams with 120 developers. RAG gives me dbCORE vibes so to say. Good luck and fun with your RAG systems.
- alex-zaporozhan 2mo agoI think so too. RAG with its layers and fine-tuning captures the imagination. Sometimes you even lose the thread between where it is math and when it is just intuitively obvious
- IronyMan1 2mo agoI believe the second Suggestion solves 95% of my problems. I want a system where i can describe my search and the system generated 5-15 keywords for a query
- comandillos 2mo agoI indexed thousands of documents into a SQLite database with an FTS5 index, plugged it into DeepSeek v4 Flash and got better much better results than any other commercial solutions my company has tried in the past. The trick was just to let the LLM come up with its own SQL queries for searching... and the results are impressive.
- ifoxhz 2mo agoI'm now using this approach too, and it feels better than any sorting method I've used before. The only thing I'm thinking about now is: if the LLM makes a mistake, how can I provide feedback and verify it?
- comandillos 2mo agoI have a web user interface connected to a coding agent (OMP) running inside a container, so if any of the tool calls fail or something happens, usually my model recovers autonomously from these situations. The capabilities of models like DS4 Flash are those of frontier models from months ago, so its recovery and autonomous capabilities are quite impressive.
- b112 2mo agoI log all toolcalls to a file, I think others have said the same. But I'm a bit leery of letting a hallucinating LLM write SQL queries. I think most I've spoken with, agree that an LLM is like a 20 year old, eager intern. Well meaning, but left unrestrained capable of immensely inexperienced mistakes. Before a lot of frameworks existed, you'd see DEVs taking user input on a web form, and then just throwing it directly at the MTA. So spammers could submit email@address\nCC: persontospam@address, and the like. Now LLMs are a different beast, but you have input validation for LLMs, unique to all other validation methods. Yet there's actually no safe way to ever validate user input for a LLM, except for very rigid input validation on single words. Take the email example above. You'd need a regex to only validate an email address (and that isn't simple), but once you expand it to actually allowing sentences? The LLM is now input validation vulnerable. And that means no user input can be used in unvalidated commands. And then just random hallucinations. I'm curious how the gp managed weirdo LLM behaviour, like out of the blue 'drop table' or accidental select into as opposed to just select.
- deleted 2mo ago[deleted]
- deleted 2mo ago[deleted]
- MarkMarine 2mo agoThis depends on your users and doc corpus. On finance docs and with users that use jargon and acronyms heavily, BM25 can fail on trivial queries... you end up encoding a masters in economics into the query re-writing logic. Double hard if you're dealing with private market finance customers with their own ideas on what "common" terms mean. I tried to replace the embedding/ingestion pipeline multiple times and nothing I tried was better over a large amount of documents. Performance sucked, the agent was re-writing and re-trying queries over and over until it found what it wanted, and vector search with a little work up front was worlds better (though it was expensive)
- rao-v 2mo agoA frontier LLM based query rewriter has atleast a masters in economics, and a pretty good understanding of finance informal language. How long ago did you try this? I'd be curious if you find this still to be the case.
- j0selit0 2mo agoI work on a really similar scenario. We ended up having to create a so-called "semantic layer" containing metadata (table and schema descriptions) and glossary terms. Still, there is a lot of work involved maintaining glossary terms since some of them are ambiguous and people have different understanding/interpretations for some of them.
- kvivek05 2mo agohave you also explored graph based mechanisms to bypass some of these challenges - especially around noise generated due to similarity?