4 ms·
> screwed up their tokenizer in FLAN-T5 so it skipped Polish diacritics. How is that even possible? Shouldn't byte-pair encoding find and encode all the diacri
by Karellen 2y ago
> screwed up their tokenizer in FLAN-T5 so it skipped Polish diacritics.
How is that even possible? Shouldn't byte-pair encoding find and encode all the diacritics (along with common digraphs and the like) as one of the first things that happens in the training process for a language?