3 ms·
It is less byte pair at that point (as the indic languages generally take up 2-3 bytes per codepoint.)
by samarthr1 3y ago
It is less byte pair at that point (as the indic languages generally take up 2-3 bytes per codepoint.)
- lifthrasiir 3y agoYeah, of course, but I thought BPE should pick it up anyway if your training data is primarily written in Indic scripts.