4 ms·
Wow. 1 million token length.
by sidcool 3y ago
Wow. 1 million token length.
- Alifatisk 3y agoYeah this is huge, first Gemini and now Claude!
- glenstein 3y agoRight, and it's seems very doable. We've been getting little bells and whistles like "custom instructions" have felt like marginal addons. Meanwhile huge context windows seem like they are a perfect overlap of (1) achievable in present day and (2) substantial value add.
- FergusArgyll 3y agoHow did everyone solve it at the same time and there is no published paper (that I'm aware of) describing how to do it? It's like every AI researcher had an epiphany all at once
- tempusalaria 3y agoFirms are hiring from each other all the time. Plus there’s the fact that the base pertaining is being done at higher context lengths, so then the context extending fine tuning is working from a larger base
- fancyfredbot 3y agoA paper describing how you might do it published in December last year. The paper was "Mamba: Linear-Time Sequence Modeling with Selective State Spaces". To be clear I don't know if Claude and Gemini actually use this technique but I would not be surprised if they did something similar: https://arxiv.org/abs/2312.00752 https://arxiv.org/abs/2312.00752 https://github.com/state-spaces/mamba https://github.com/state-spaces/mamba