3 ms·

I kid you not, people at Google (or HuggingFace) screwed up their tokenizer in FLAN-T5 so it skipped Polish diacritics. This ain't a rare language either. As an

I kid you not, people at Google (or HuggingFace) screwed up their tokenizer in FLAN-T5 so it skipped Polish diacritics. This ain't a rare language either. As an aside, the model still managed to sidestep the issue most of the time.

Karellen 3y ago
> screwed up their tokenizer in FLAN-T5 so it skipped Polish diacritics. How is that even possible? Shouldn't byte-pair encoding find and encode all the diacritics (along with common digraphs and the like) as one of the first things that happens in the training process for a language?