4 ms·

Any good search you used before all this LLM stuff started happening is a perfect candidate for RAG. How do you know if a search was good? If you weren't pull

Any good search you used before all this LLM stuff started happening is a perfect candidate for RAG. How do you know if a search was good? If you weren't pulling your hair out and actually got decent results for your queries (search is a thankless job like that - everyone expects it to work and complains when it doesnt). The reason good search is best for RAG is because the prompt is seeded by the top results for the query. The only thing RAG does is summarize things for you and gives you answers instead of a list of documents. And now I gotta confess something, after making RAG systems for clients and having to use them with all the web search engines these days - I kinda miss the list of documents, and find myself just skipping the summary at the top half the time and going back to reading the 10 blue links.

natsucks 3y ago
Interesting. Do you think that points to the current limitations of RAG or a mismatch in what a user truly wants from search?
- binarymax 3y ago
  I think it works well when it's not a blob of text. One issue is that most of them are really long-winded. For example, if the answer can be nouns, just give me the list of nouns instead of a full sentence or paragraph. Take for example this search: https://search.brave.com/search?q=what+are+the+captain+america+movies%3F&source=web https://search.brave.com/search?q=what+are+the+captain+ameri... Why the paragraph? Just give me a bulleted list! It's hard to read and kinda annoying. Another issue for me is trust. Web search is oft polluted with web spam (this is not new). Mentally, one can see a URL and skip a site that doesn't have strong authority. So now in RAG, I either need to trust the answer, or I need to look at the embedded citation and find the document and then see if it's trustworthy. This adds friction. This is also not unique to web search. Private search can also have poor relevance - do I know the LLM is being given the best context? Or is it getting bad context and hallucinating? I need to look at the results to be sure anyway. I think when used in appropriate ways it can be good. But the experience of "summarize these 10 results for me" might not be the best for every query.
  - DebtDeflation 3y ago
    You're referring to what in the NLP subfield of Question Answering Systems would be known as a "factoid question". Historically, things like knowledge graphs and RDF triple stores would be used for answering these types of questions. I'm still not sold on the idea that an LLM is the answer to all QA/Chat problems and this is one example.

natsucks 3y ago

Interesting. Do you think that points to the current limitations of RAG or a mismatch in what a user truly wants from search?

binarymax 3y ago
I think it works well when it's not a blob of text. One issue is that most of them are really long-winded. For example, if the answer can be nouns, just give me the list of nouns instead of a full sentence or paragraph. Take for example this search: https://search.brave.com/search?q=what+are+the+captain+america+movies%3F&source=web https://search.brave.com/search?q=what+are+the+captain+ameri... Why the paragraph? Just give me a bulleted list! It's hard to read and kinda annoying. Another issue for me is trust. Web search is oft polluted with web spam (this is not new). Mentally, one can see a URL and skip a site that doesn't have strong authority. So now in RAG, I either need to trust the answer, or I need to look at the embedded citation and find the document and then see if it's trustworthy. This adds friction. This is also not unique to web search. Private search can also have poor relevance - do I know the LLM is being given the best context? Or is it getting bad context and hallucinating? I need to look at the results to be sure anyway. I think when used in appropriate ways it can be good. But the experience of "summarize these 10 results for me" might not be the best for every query.
- DebtDeflation 3y ago
  You're referring to what in the NLP subfield of Question Answering Systems would be known as a "factoid question". Historically, things like knowledge graphs and RDF triple stores would be used for answering these types of questions. I'm still not sold on the idea that an LLM is the answer to all QA/Chat problems and this is one example.