2 ms·
Isn't that a specialization of BPE for specific distribution? BPE is essentially a data compression scheme that constructs a straight-line grammar (SLG) that ca
by lifthrasiir 3y ago
Isn't that a specialization of BPE for specific distribution? BPE is essentially a data compression scheme that constructs a straight-line grammar (SLG) that can be directly used as a model encoding, and it is hardly surprising that any other SLG would work well for some other inputs.
- samarthr1 3y agoIt is less byte pair at that point (as the indic languages generally take up 2-3 bytes per codepoint.)
- lifthrasiir 3y agoYeah, of course, but I thought BPE should pick it up anyway if your training data is primarily written in Indic scripts.