2 ms·the new tokenizer (in 4.7+ iirc) produces about 30% more tokens on averageby verdverm 5mo agothe new tokenizer (in 4.7+ iirc) produces about 30% more tokens on average