I am not really disagreeing with you, but just wanted to add: Probably the secret sauce to using LLMs for search and other forms of information retrieval is fi

I am not really disagreeing with you, but just wanted to add: Probably the secret sauce to using LLMs for search and other forms of information retrieval is first taking a query and use traditional lookup to find likely information. Then aggregating this data, provide the aggregated data as context, append the query, and pass it to the LLM. Probably some post processing is a good idea also. I used to do this with squad LLMs.