3 ms·
Somewhat confusingly, it appears the tokenizer vocabulary as well as the context length are both 128k tokens!
by sva_ 2y ago
Somewhat confusingly, it appears the tokenizer vocabulary as well as the context length are both 128k tokens!
- littlestymaar 2y agoYup, that's why I wanted to clarify things.