3 ms·

>Farm it out to low-paid non-native speakers The paper claims: >“All raters are native English speakers from India.”

pmontra 4y ago
US English != Indian English especially if you have to actually know the cultural details behind some sentences. I bet US native speakers would have similar failure rates at labeling English sentences from Indian media, because they belong to different cultures.
- nmfisher 4y ago
  The blog post highlights this specific point - "US English" and "Indian English" really aren't the same English (in fact, I'd probably go even further and state that "Reddit English" and "US English" probably aren't the same English either). Likewise, the Common Voice English dataset isn't great for ASR training outside India, either. There's a huge proportion of Indian speakers, and their data doesn't really help train ASR systems for non-Indian accents.
- david_draco 4y ago
  Is the "right answer" a classification based on US English with background knowledge of US cultural background, or is the goal to build a global sentiment data set? You and OP ("the indians labelers don't know how to do it correctly") seem to want the former, so it would be good to state that goal upfront.
  - zharknado 4y ago
    I think it’s valid to question US-centrism broadly in datasets like this, but presumably that choice was made in selecting the Reddit sample, which is dominated by North American English. A model is more useful if it approximates the speaker’s intent, not the listener’s interpretation. “Right” is complicated in language, but it’s hard to see how you’d use a model full of cross-cultural misunderstandings.
bryanrasmussen 4y ago
https://www.heritagexperiential.org/language-policy-in-indian-schools-a-point-of-view/ https://www.heritagexperiential.org/language-policy-in-india... >English, due to its ‘lingua franca’ status, is an aspiration language for most Indians – for learning English is viewed as a ticket to economic prosperity and social status. Thus almost all private schools in India are English medium. Many public schools, due to political compulsions, have the state’s official languages as the primary school language. English is introduced as a second language from grade 5 onwards.
fluidcruft 4y ago
Perfectly logical choice if you're building a machine to replace call centers.

US English != Indian English especially if you have to actually know the cultural details behind some sentences. I bet US native speakers would have similar failure rates at labeling English sentences from Indian media, because they belong to different cultures.

nmfisher 4y ago
The blog post highlights this specific point - "US English" and "Indian English" really aren't the same English (in fact, I'd probably go even further and state that "Reddit English" and "US English" probably aren't the same English either). Likewise, the Common Voice English dataset isn't great for ASR training outside India, either. There's a huge proportion of Indian speakers, and their data doesn't really help train ASR systems for non-Indian accents.
david_draco 4y ago
Is the "right answer" a classification based on US English with background knowledge of US cultural background, or is the goal to build a global sentiment data set? You and OP ("the indians labelers don't know how to do it correctly") seem to want the former, so it would be good to state that goal upfront.
- zharknado 4y ago
  I think it’s valid to question US-centrism broadly in datasets like this, but presumably that choice was made in selecting the Reddit sample, which is dominated by North American English. A model is more useful if it approximates the speaker’s intent, not the listener’s interpretation. “Right” is complicated in language, but it’s hard to see how you’d use a model full of cross-cultural misunderstandings.

bryanrasmussen 4y ago

https://www.heritagexperiential.org/language-policy-in-indian-schools-a-point-of-view/ https://www.heritagexperiential.org/language-policy-in-india... >English, due to its ‘lingua franca’ status, is an aspiration language for most Indians – for learning English is viewed as a ticket to economic prosperity and social status. Thus almost all private schools in India are English medium. Many public schools, due to political compulsions, have the state’s official languages as the primary school language. English is introduced as a second language from grade 5 onwards.

fluidcruft 4y ago

Perfectly logical choice if you're building a machine to replace call centers.