3 ms·
It would be inefficient if you had to do it from scratch for every query, but if you can do it once as a preprocessing step and reuse the prepared context for m
by yorwba 1y ago
It would be inefficient if you had to do it from scratch for every query, but if you can do it once as a preprocessing step and reuse the prepared context for many queries, it might start to become more efficient than a shorter context that includes only some documents but has to be reprocessed because it's different every time.
- killerstorm 1y agoYes, I think it might be a good solution where you have a context up to 10M of tokens and you do a lot of requests with that context. It might be relevant for agentic stuff which tends to produce long chat logs - especially with some gadgets on top, e.g. some 'episodes' might be completely removed as obsolete. But I don't think it's a good solution for bigger amounts of data - as in that case it's more beneficial if that can be formed into independent memories.