3 ms·
In the Indian subcontinent, most native content features a lot of code-mixing - so in a Bengali or Hindi document you'll see some English words for sure. And t
by shriphani 7y ago
In the Indian subcontinent, most native content features a lot of code-mixing - so in a Bengali or Hindi document you'll see some English words for sure.
And the vast majority of native content is written in the Roman script - (native script keyboards are poorly designed or unavailable I suppose).
Thus a large chunk of content gets tossed in as English - granted it won't be a high confidence prediction but it still is the produced label.
Corpora in the subcontinent can manifest in 10 - 12 languages - say the Rohingya language for instance - it is near impossible to get an annotator that speaks that language. Getting a monolingual corpus out with zero annotation is quite useful.