3 ms·
This is no surprise, the tokenizer is constructed to minimize the encoded length of the training corpus, and most of that is in English or at least using the La
by ruuda 3y ago
This is no surprise, the tokenizer is constructed to minimize the encoded length of the training corpus, and most of that is in English or at least using the Latin alphabet.