3 ms·
I'm reminded of how often I see Twitter offer to translate English-language tweets containing a proper noun or two from absurdly unconnected languages. And that
by notahacker 9y ago
I'm reminded of how often I see Twitter offer to translate English-language tweets containing a proper noun or two from absurdly unconnected languages. And that's with text containing mostly common and distinctive English words.
- abecedarius 9y agoI'm pretty sure Twitter's langid uses character n-grams. You'll see a tweet that's plain English that happens to match n-gram statistics unusually well with some other language, which pushes the likelihood score to just above English. (I checked this by running an example or two through my own langid code.) It shouldn't be hard to improve on by treating the score on a tweet as bayesian evidence to combine with a prior from preceding tweets.