5 ms·
Unless there is a significant increase in the effective context window in LLMs. Pursuing the goal of having agents working on complex goals is not going to work
by elashri 2y ago
Unless there is a significant increase in the effective context window in LLMs. Pursuing the goal of having agents working on complex goals is not going to work well. All the tricks and hacks trying to work around this problem is not going to fundamentally change that.
LLM agents will lose track of what they are trying to do after couple of trials. That's something that would differentiate human PhD is that while not fast or always creative, they have better attention memory span.
- xnx 2y agoWhat's would be a significant increase to you? Gemini does 2 million tokens and Google just released a research paper to go beyond that.
- OutOfHere 2y agohttps://github.com/MiniMax-AI/MiniMax-01 https://github.com/MiniMax-AI/MiniMax-01 is an open model that claims a 4 million context. Note however that longer context makes evaluation expensive as you are paying for every token. Still, it is true that OpenAI seriously needs a better solution for it.
- AstralStorm 2y agoHow many tokens in a Bible? A typical technical book? Thing will lose context before it has finished reading...
- OutOfHere 2y agoI think it's time to partition the context into L1, L2, and L3 contexts. L1 is the current context with a quadratic memory requirement. L2 is based on fancy mechanisms such as what is used by Gemini and MiniMax-01, having a sub-quadratic to linear memory requirement. L3 is based on document and chunk embeddings having a linear to logarithmic memory requirement. LLMs don't use this approach, but I think it might make sense. As for how this partitioning would work at the neural layers, that remains to be determined.
- OutOfHere 2y agoTitans [1] goes sort of a bit in this direction. [1] https://arxiv.org/abs/2501.00663 https://arxiv.org/abs/2501.00663
- mountainriver 2y agoReasoning is probably enough