43 ms·
> as every time you make a call, the system loads a large chunk or many chunks and sends them to the model along with your prompt, This is how RAG works. Whil
by nl 2y ago
> as every time you make a call, the system loads a large chunk or many chunks and sends them to the model along with your prompt,
This is how RAG works.
While you can come up with work-arounds like using lesser LLMs as a pre-filtering step the fact is that if you need GPT to read the doc you need GPT to read the doc.
- firejake308 2y agoTrue, this is how RAG works, but this is why I prefer to use open-source LLMs for RAG: because the token costs are less opaque and I can control how many chunks I pull fromthe database to manage my costs
- yatz 2y agoI believe it will get better and more efficient as we go. On a side note, OpenAI seems to release products before they are ready and they evolve as they go.
- nl 2y ago> I believe it will get better and more efficient as we go. Yes of course. The point remains: the LLM has to process the data somehow. If you are concerned about costs and token usage then switch to a provider that works for your problem (Flash Gemini looks very interesting..)