4 ms·
Mixtral 8X7B has a 32k-token context window. GPT-3.5 models have a context window of 4k-16k. As for the sliding window attention, the model does not lose all t
by rockinghigh 3y ago
Mixtral 8X7B has a 32k-token context window. GPT-3.5 models have a context window of 4k-16k. As for the sliding window attention, the model does not lose all the information about the tokens before the sliding window. Hidden states store information about past tokens.
- ComputerGuru 3y agoThe sliding window does not lose the data but it does "decompose" it so that it can't be recalled verbatim. For analyzing code (feed it n classes and ask it to create a class using all of them to accomplish a task) that isn't good enough. It's also not good enough for some of the "corporate business" use cases we tried putting Mistral and other sliding window models to use on, where you need it to re-use verbatim or reference specific portions of one of n documents fed into it as input tokens. Again, sufficient training can overcome these limitations. But that's only for cases where the corpus of input documents is static or at least contains significant reuse.