> RAG is quite the hack. All you start with is the question, but why should the documents you need for the answer be similar to the question? I have always tho

> RAG is quite the hack. All you start with is the question, but why should the documents you need for the answer be similar to the question? I have always thought that a better RAG system would be to let the LLM use the search in the same way that a human would - think about good keywords and search the needed information iteratively until everything needed is retrieved. I have even wrote a proof of concept: https://github.com/zby/answerbot https://github.com/zby/answerbot and wrote a blog post (mostly) about that approach: https://zzbbyy.substack.com/p/design-principles-for-llm-based-systems https://zzbbyy.substack.com/p/design-principles-for-llm-base... But then I discovered that it is very hard to tell the LLM to both use its reasoning and the search - for example if I ask GPT-4 "What is the weight proportion of oxygen in water? Think step by step" it lays out a very good thought process that takes the facts that it knows about the chemical formula of water, the atomic weights of hydrogen and oxygen and does the math to arrive at the good answer. But if I just add wikipedia search functions to the llm call (https://github.com/zby/answerbot/blob/ea792fbc7a84c6683dc2f37a7880b726fff70558/react.py#L10 https://github.com/zby/answerbot/blob/ea792fbc7a84c6683dc2f3...) even without any additional prompting or examples in the prompt it starts searching and rather blindly - it stops any reasoning and tries to find the final answer in wikipedia instead of looking for the intermediate information - like the chemical formula, the atomic weights. I guess I'll find a workaround that problem with some clever prompting - but this difficulty surprised me.