4 ms·
This is one of the most confusing claims I've seen in a long time. Grep and others over files would be the equivalent of an old fashioned keyword search where m
by tensor 6mo ago
This is one of the most confusing claims I've seen in a long time. Grep and others over files would be the equivalent of an old fashioned keyword search where most RAG uses vector search. But everything else they claim about a file system just suggests that they don't know anything about databases.
I'm not familiar with how most out of the box RAG systems categorize data, but with a database you can index content literally in any way you want. You could do it like a filesystem with hierarchy, you could do it tags, or any other design you can dream up.
The search can be keyword, like grep, or vector, like rag, or use the ranking algorithms that traditional text search uses (tf-idf, BM25), or a combination of them. You don't have to use just the top X ranked documents, you could, just like grep, evaluate all results past whatever matching threshold you have.
Search is an extremely rich field with a ton of very good established ways of doing things. Going back to grep and a file system is going back to ... I don't know, the 60s level of search tech?
- brap 6mo agoI get what you’re saying, and you’re right, however I can also see where they’re coming from: Empirically, agents (especially the coding CLIs) seem to be doing so much better with files, even if the tooling around them is less than ideal. With other custom tools they instantly lose 50 IQ points, if they even bother using the tools in the first place.
- tensor 6mo agoSorry, this still makes no sense. LLMs don't care about files. The way most codings systems work is that they simply provide the whole file to the LLM rather than a subset of it. That's just a choice in how you implemented your RAG search system and database. In this case the "record" is big, a file. No doubt that works for code, but it's nonsensical outside that. E.g. for wikipedia the logical unit would likely be an article. For a book, maybe it's a chapter, or maybe it's a paragraph. You need to design the system around your content and feed the LLM an appropriate logically related set of data.
- darkteflon 6mo agoYeah, some of the uplift people are anecdotally seeing from “just using the filesystem” is, imo, on account of how difficult it is to take a principled approach to pre-chunking when implementing other approaches.
- brap 6mo ago>LLMs don't care about files. Oh but they do. These CLI agents are trained and specifically tuned to work with the filesystem. It’s not about the content or how it’s actually stored, it’s about the familiar access patterns. I can’t begin to tell you how many times I’ve seen a coding agent figure out it can get some data directly from the filesystem instead of a dedicated, optimized tool it was specifically instructed to use for this purpose. You basically can’t stop these things from messing with files, it’s in their DNA. You block one shell command, they’ll find another. Either revoke shell access completely or play whackamole. You cannot believe how badly they want to work with files.
- girvo 6mo agoThey've been RLHF'd to the nth degree around working with *nix tools and filesystems, in practice.
- pertymcpert 6mo agoThey do care about files. They also care about how you express yourself, your tone, all sorts of seemingly unimportant details.
- raincole 6mo ago> LLMs don't care about files They do. I highly suggest not try to derive LLMs' behaviors (in your mind) from first principles, but actually use them.
- pjm331 6mo agoYeah I’ve had a lot of success with agentic search against a database. The way I think of it, the main characteristic of agentic search is just that the agent can execute many types of adhoc queries It’s not about a file system As I understood it early RAG systems were all about performing that search for the agent - that’s what makes that approach “non agentic” But when I have a database that has both embeddings and full text and you can query against both of those things and I let the agent execute whatever types of queries it wants - that’s “agentic search” in my book
- darkteflon 6mo agoAbsolutely, agentic search is much more robust to the specific implementation details of your search setup (data quality issues, too) than the early one-shot approaches were. Anyone watching Claude Code work can see this for themselves.
- thefourthchime 6mo agoI didn't get into the details too much, but I kept thinking, why isn't he just having an agent discover things from various data sources? I've had much better success with that.
- deleted 6mo ago[deleted]
- jimbokun 6mo agoIsn’t this the approach described in the article?
- dboreham 6mo agoAlso odd in that most filesystems implement directories and file names as...a database. You can use a filesystem as a database but you're not being as clever as you thought.