3 ms·
This seems like an argument for a character based rather than word based network. It just so happens that English and Chinese, the two languages which have the
by kmicklas 10y ago
This seems like an argument for a character based rather than word based network. It just so happens that English and Chinese, the two languages which have the most machine learning research, are relatively analytic, with a low morpheme per word ratio. But many world languages have a much higher ratio (Russian wouldn't even rank that high!) and acquiring training data covering all unique "words" is essentially impossible.
- srush 10y agoI'm glad you mentioned this. There is a lot of interest these days in character-based machine translation, including several papers in review at ICLR. The current practical consensus (at least in OpenNMT) is that character-only models are not really worth the efficiency loss. A simple compromise is to use Byte-Pair Encoding as a preprocessing step in morphologically rich languages and allow the model to produce sub-word chunks. This is implemented in OpenNMT as a preprocessing option (see http://opennmt.net/Advanced http://opennmt.net/Advanced).