> well indexed documentation Indices are, by definition, lossy representations of their underlying data. If you use stemming and lemmatization to preprocess bo

> well indexed documentation Indices are, by definition, lossy representations of their underlying data. If you use stemming and lemmatization to preprocess both documentation and query text, you're already departing from a truly hand-optimized indexing system, and choosing to have imperfect algorithms do things in a more scalable way. And indexing by embedding vectors that use LLMs to determine context are a natural extension of this, in my view. And on top of that, when you have a massive amount of candidate text to display to the user... is displaying sentence fragments one on top of the other the most optimal UX there? At a certain point, RAG becomes the answer to this question. The problem, as you note, is that search engines and social media systems are incentivized to allow garbage content into the original set of things they index and surface, if that garbage content drives more attention to advertisements. But that's not a reason to reject the benefits that the underlying LLM technology can bring towards building good indexing on top of human-written documents. It just won't be done by the companies that used to do it.