2 ms·

I think that modern BPE and techniques are more than sophisticated enough to handle things like this.

• m0zg 6y ago
  Could be, but I've yet to see an example of any NLP task that performs anywhere near as well on Russian as it does on English. Trouble with NLP in general, is that humans, armed with their imperceptible error correction and general intelligence, expect unreasonable things from systems which fundamentally have neither. Error rate in single digit percentages is commonly perceived as terrible, and because of the cardinality of the input space "plausible" errors don't really happen and failure modes are mostly catastrophic. This applies to all languages BTW. There are only very, very few NLP systems that actually work, and they are all in a setting which tolerates a fairly high error rate (e.g. ranking).
  • MiroF 6y ago
    > any NLP task that performs anywhere near as well on Russian as it does on English This has a lot more to do with smaller Russian corpora than anything to do with peculiarities of the Russian language. > This applies to all languages BTW. There are only very, very few NLP systems that actually work, and they are all in a setting which tolerates a fairly high error rate (e.g. ranking). I've trained NLP models used on over 15 different languages used in production