3 ms·

Humans that read (at least) Indo-European languages can read texts in their native language with all the vowels removed. Does that suggest that it would be a go

Humans that read (at least) Indo-European languages can read texts in their native language with all the vowels removed. Does that suggest that it would be a good idea to remove the vowels from text before using it for training text-based LLMs ? Presumably you want to train on as rich a set of data as possible, even if some of that data is redundant or irrelevant when it comes to human perception.

Tangurena2 1y ago
Generally, the difference between regional dialects is almost all in vowels (sample: 0). This is why SOUNDEX [1] eliminated vowels. 0 - https://www.acelinguist.com/2020/01/the-pin-pen-merger.html https://www.acelinguist.com/2020/01/the-pin-pen-merger.html 1 - https://en.wikipedia.org/wiki/Soundex https://en.wikipedia.org/wiki/Soundex