4 ms·
How many tokens in a Bible? A typical technical book? Thing will lose context before it has finished reading...
by AstralStorm 2y ago
How many tokens in a Bible? A typical technical book?
Thing will lose context before it has finished reading...
- OutOfHere 2y agoI think it's time to partition the context into L1, L2, and L3 contexts. L1 is the current context with a quadratic memory requirement. L2 is based on fancy mechanisms such as what is used by Gemini and MiniMax-01, having a sub-quadratic to linear memory requirement. L3 is based on document and chunk embeddings having a linear to logarithmic memory requirement. LLMs don't use this approach, but I think it might make sense. As for how this partitioning would work at the neural layers, that remains to be determined.
- OutOfHere 2y agoTitans [1] goes sort of a bit in this direction. [1] https://arxiv.org/abs/2501.00663 https://arxiv.org/abs/2501.00663