23 ms·
Llama 3.2 1B and 3B both claim a 128,000 token length.
by simonw 2y ago
Llama 3.2 1B and 3B both claim a 128,000 token length.
- GaggiX 2y agoThat is indeed their claim, but if you read the linked page you will see that it is not their effective token window.
- dragonwriter 2y agoThe issue isn't the nominal size of the context window which is easy to objectively measure, but effective use of the context window, which is harder to measure precisely, but a big issue: it doesn't matter how much stuff you can throw at the model if it effectively ignores most of it past a certain point.
- vessenes 2y agoThey point out in the paper stats around where effectiveness drops off hard. It's at 32k at most everywhere now. Some models still at 4 to 8k. Getting to 200k in a smaller model is an open research area, as far as I'm aware. Ideas so far are pretty broad ranging, including using fourier transforms to try and capture cyclicality in inputs (mamba et al), RWKV (which I do not fully understand, but claims very long input context support), larger range of possible tokenizations shortening input length (llama 3), ..