3 ms·
And, for indic languages, there is orthographic syllable pair encoding, which reduces the amount of space taken by indic characters by encoding the most common
by samarthr1 3y ago
And, for indic languages, there is orthographic syllable pair encoding, which reduces the amount of space taken by indic characters by encoding the most common letter pairs as a single byte.
This was presented i think last year, at the URTC in MIT.
(Disclosure, it was published by my friend)
- lifthrasiir 3y agoIsn't that a specialization of BPE for specific distribution? BPE is essentially a data compression scheme that constructs a straight-line grammar (SLG) that can be directly used as a model encoding, and it is hardly surprising that any other SLG would work well for some other inputs.
- samarthr1 3y agoIt is less byte pair at that point (as the indic languages generally take up 2-3 bytes per codepoint.)
- lifthrasiir 3y agoYeah, of course, but I thought BPE should pick it up anyway if your training data is primarily written in Indic scripts.
- mistrial9 3y ago> for indic languages amateur here - but "Indic languages" are quite different. Maybe you mean Hindi and that group ?
- ubutler 3y agoPresumably, they’re referring to the Indic scripts (also known as Brahmic scripts): https://en.wikipedia.org/wiki/Brahmic_scripts https://en.wikipedia.org/wiki/Brahmic_scripts
- samarthr1 3y agoI am! I mean to say the languages usually represented in brahmic scripts (With most of the brahmic scripts being phonetic, one can write multiple languages in the same script in a sensible manner)