On the subject of cost. I looked at how they do Relevance Extraction, I.e., extract portions of a passage that are relevant to a query (a critical step in a RAG

On the subject of cost. I looked at how they do Relevance Extraction, I.e., extract portions of a passage that are relevant to a query (a critical step in a RAG pipeline), and they do it the obvious way, I.e have the LLM parrot out the relevant sentences one by one (“Parrot” not inspired by their logo I swear;). Before looking at their LLMChainExtractor that does this, I had already implemented this massively more efficiently by pre-annotating the passage with sentence numbers and having the LLM simply tell you the relevant sentence numbers. It was natural to do this in an agent loop with function-calling in the Langroid[1] framework. This is both faster (since it has to generate very few tokens) as well as cheaper (remember token generation cost is double the input cost when using GPT4 at least). I was actually surprised that LangChain doesn’t do it this way. Just an example of how we shouldn’t assume the established implementations are the best ones and one should always be skeptical and take a fresh look. I posted about this a couple days ago— https://www.linkedin.com/posts/pchalasani_rag-llm-langchain-activity-7124823472869953537-9T1T https://www.linkedin.com/posts/pchalasani_rag-llm-langchain-... [1] Langroid: https://github.com/langroid/langroid https://github.com/langroid/langroid