2 ms·
Does the encoding have anything to do with it? When talking about token length, I couldn't help but wonder if they were judging length in UTF-8 bit size, in wh
by pcthrowaway 3y ago
Does the encoding have anything to do with it?
When talking about token length, I couldn't help but wonder if they were judging length in UTF-8 bit size, in which case languages using non-Latin alphabets (and even those that do, but with accents) would pay a penalty