2 ms·
The problem is that detecting the different variants to label the training data requires understanding the semantics (detecting French vs English can be done pu
by zarzavat 3y ago
The problem is that detecting the different variants to label the training data requires understanding the semantics (detecting French vs English can be done purely lexically), which has historically been a hard problem. With modern language modeling it’s tractable.
- goblin89 3y agoI think Google has the resources required to detect different languages in Chinese script (different combinations of characters and vocabulary, even specific Unicode characters, etc.) but there is 1) lack of incentives and 2) opposing political interests. Doesn’t help that there is a bit of a mess when it comes to Chinese characters, like a single Unicode point can be rendered differently depending on the font.