7 ms·
"Approximately 8 of the 319 million people in the United States read the Wall Street Journal, about 2 percent of the population. If you look at the language — s
by xiaoma 10y ago
"Approximately 8 of the 319 million people in the United States read the Wall Street Journal, about 2 percent of the population. If you look at the language — standardized English — being fed into many natural language processing units, it’s based on the language of that 2 percent. "
It's hard to take an article seriously when it opens with a logical fallacy. Yes, the WSJ uses standard English and only 8 million people read (subscribe to?) it. That does not mean that only 8 million people in the US use so-called standard English. There are also a lot of people who don't subscribe to that particular publication but still speak or write in "standard" English and that group is much larger than the group of WSJ readers.
- mattdeboard 10y agoNah. It'd be wrong in the way you suggest if they said it's the language of only that 2%, I think. It IS "the" language of that two million. Just not ONLY those two million.
- lqdc13 10y agoThen why bring up WSJ at all? What about all the other papers? and what about all the other sources of training data for NLP research? WSJ dataset is not the only such dataset. NLP is not very good with standard English yet and usually doesn't generalize from topic to topic. Dialects and other languages - especially those without formal rules - will come when we can deal with standard English.
- gordonguthrie 10y agoAll language has rules - there is no language 'without formal rules'
- morgante 10y agoWhat are the "formal rules" of AAVE?
- thaumasiotes 10y agohttps://en.wikipedia.org/wiki/African_American_Vernacular_English#References https://en.wikipedia.org/wiki/African_American_Vernacular_En...
- morgante 10y agoI'm looking for formal rules. I'm not saying AAVE doesn't exist, I'm saying its rules (such as they are) are not formalized. Tons of rule books exist for Standard English and part of learning it is to memorize the correct rules. I don't believe any formal equivalent exists for AAVE.
- iamdave 10y agoWhat, then, are your parameters for formal? It has to be in a book, codified and signed off on? If so, aren't these kind of antithetical to AAVE on the face of it?
- morgante 10y ago> It has to be in a book, codified and signed off on? The point is for rules to be formal then the must be formalized somehow and codified. This could be online or in a book, but the point is that there must be some clear delineation between when the rules are followed and when they are broken. Otherwise what's the meaning of "formal" rules? > If so, aren't these kind of antithetical to AAVE on the face of it? Yes. My argument is that AAVE, almost by definition, is an informal dialect without formalized rules. This is getting rather off topic, but I think it relates to the original point of why NLP might start with Standard English even if you are not biased. A large corpus of Standard English text (such as from the WSJ) will generally be very internally consistent precisely because it follows a set of formal rules codified into a style guide. As there is no such equivalent for AAVE, even gathering a large and internally consistent corpus of AAVE text seems prohibitively difficult. That being said, I do hope researchers are working on gathering text from Twitter to build up new training sets.
- tedunangst 10y ago"I read the Wall Street Journal, about 0.00000031 percent of the population. If you look at the language — standardized English — being fed into many natural language processing units, it’s based on the language of that 0.00000031 percent." Equally true, right? But somehow it seems just a little misleading.
- Dylan16807 10y agoYou can read the WSJ without going about your everyday life forming sentences the same way. It's a terrible claim no matter how you interpret it.
- ubernostrum 10y agoIt's hard to take a poster seriously when they show willful ignorance of the way language serves as a marker of class, ethnicity and race, and instead have to manufacture quibbles to try to discredit an article that might cause slightly less comfort with the notion of the Glorious AI Future™ -- Coming Soon™!
- Dylan16807 10y agoI think your post is the most aggressive instance of putting strawman words in someone else's mouth I've seen all month.
- briholt 10y ago> When researchers evaluated their model against natural language processing tools...researchers found that the software flagged African-American English as “not English” at a much higher frequency than standard English. Likewise, notice the deceptive switch between "English" and "standard English." So basically language bots trained to identify standard English are correctly identifying non-standard English dialects as not standard English.