For those who interested, there some new researches in the field [0]. It usually possible to create more compact token representation from given text, but my gu

For those who interested, there some new researches in the field [0]. It usually possible to create more compact token representation from given text, but my guess the greedy "optimal" tokenizer might harm the performance of the model? [0] https://www.reddit.com/r/LocalLLaMA/comments/140gcn7/new_tokenizer_increases_inference_speed_and/ https://www.reddit.com/r/LocalLLaMA/comments/140gcn7/new_tok...