4 ms·
Does anyone know what changed in the tokenizer? Does it output multiple tokens for things that were previously one token?
by ai_slop_hater 6mo ago
Does anyone know what changed in the tokenizer? Does it output multiple tokens for things that were previously one token?
- quux 6mo agoIt must, if it now outputs more tokens than 4.6's tokenizer for the same input. I think the announcement and model cards provide a little more detail as to what exactly is different