3 ms·Turning any tokenizer into a greedy one2 points by stephantul 1y agostephantul 1y agoI recently did some work on making tokenizers greedy.