5 ms·

Is this a result of the AI being written by a bunch of non-black people? It seems to me that this problem is more of a reflection of the people working on the c

Is this a result of the AI being written by a bunch of non-black people? It seems to me that this problem is more of a reflection of the people working on the code.

a3n 10y ago
There may be some of that, but I think it has more to do with AI still being young, and it's easier to work with less ambiguous text for learning. That and everyday speech move much faster than formal writing. Now that my son has gone off to college and is less accessible as a resource, I expect to almost totally lose touch with the speech of "these kids today."
- ralfd 10y ago
  > I expect to almost totally lose touch with the speech of "these kids today." Is this long term really a problem though? Regression to the mean: No one cares about the specific slang of the 60s anymore. And the hip slang of today will be out of fashion in 15 years. In a similar vain the extreme T9 keyboard text message abbreviations of the youth in the 2000s did go out of fashion because of automatic spell checking and speech recognition of modern smartphones.
  - thaumasiotes 10y ago
    > Is this long term really a problem though? How long term? What's "a problem"? We know with certainty that given the passage of enough time, the speech of "these kids today" will be completely unintelligible to you.
- gugagore 10y ago
  Do you think written Standard English is less ambiguous than any other English, or written language? Because I don't see why that would be the case.
  - Dylan16807 10y ago
    >Do you think written Standard English is less ambiguous than any other English Definitely. Things are clearer when you write formally and strictly adhere to grammatical rules. When I'm speaking casually I omit words and (ab)use punctuation much differently, and while that is perfectly acceptable to do, it is more complex and ambiguous to parse. As far as I'm aware the same pattern exists in a lot of languages and dialects, and it's possible to define formal AAVE too.
    - deleted 10y ago
      [deleted]
    - gugagore 10y ago
      I don't think there is anything "formal" about Standard English. Can you give an example of what you're talking about? Which grammatical rules do you have in mind? I don't know what you mean by formal black English (AAVE) vs. ... casual black English?
    - Dylan16807 10y ago
      To your comment that was detected as a duplicate: >I don't think there is anything "formal" about Standard English. "Standard" English as she is spoke? No. But a3n was specifically talking about the formal version often called "Standard written English". It is very prescriptive and less flexible, which makes it easier to parse. > Can you give an example of what you're talking about? Which grammatical rules do you have in mind? Let's start with "every rule involving punctuation".
      - gugagore 10y ago
        I'm not trying to be dense, but I still don't see it. Punctuation helps with identifying boundaries of clauses, whether they're dependent or not, whether a sentence is a question. Is this what NLP struggles with? I don't think so. One of the big unsolved problems is understanding the referents of pronouns (and other proforms, like "I will [do so] too"). Style guides and generally prescriptivism does not help that. Another ambiguity is parse ambiguities like " I saw the girl with the telescope." What does "formal" English say about that?
deleted 10y ago
[deleted]
TazeTSchnitzel 10y ago
Partly that, but my guess would be that it's because it's largely trained from written English (not the sort you see on Twitter and in SMS), and written English is usually one of the standard varieties, and the standard varieties bear little resemblance to AAVE. I would guess the same problem occurs with other dialects which are sufficiently different from the standard.
viraptor 10y ago
I expect it's mostly a reflection on the learning data in language processing. Almost any corpus of text you can easily find for experimenting is close to standard English. Dialects very rarely appear in published texts unless it's just to point out that a specific person being quoted uses that dialect. Choose a student of any colour. When they need a text corpus for research, do you think they'll reach for a different, known, large text collection that matches their background?
leftpad 10y ago
This is absolutely one of the reasons. See Google's computer vision system classifying black folks as gorillas: http://blogs.wsj.com/digits/2015/07/01/google-mistakenly-tags-black-people-as-gorillas-showing-limits-of-algorithms/ http://blogs.wsj.com/digits/2015/07/01/google-mistakenly-tag...
gjolund 10y ago
Do you know of any AI's being written by predominately black teams?

Is this a result of the AI being written by a bunch of non-black people? It seems to me that this problem is more of a reflection of the people working on the code.

a3n 10y ago

There may be some of that, but I think it has more to do with AI still being young, and it's easier to work with less ambiguous text for learning. That and everyday speech move much faster than formal writing. Now that my son has gone off to college and is less accessible as a resource, I expect to almost totally lose touch with the speech of "these kids today."

ralfd 10y ago
> I expect to almost totally lose touch with the speech of "these kids today." Is this long term really a problem though? Regression to the mean: No one cares about the specific slang of the 60s anymore. And the hip slang of today will be out of fashion in 15 years. In a similar vain the extreme T9 keyboard text message abbreviations of the youth in the 2000s did go out of fashion because of automatic spell checking and speech recognition of modern smartphones.
- thaumasiotes 10y ago
  > Is this long term really a problem though? How long term? What's "a problem"? We know with certainty that given the passage of enough time, the speech of "these kids today" will be completely unintelligible to you.
gugagore 10y ago
Do you think written Standard English is less ambiguous than any other English, or written language? Because I don't see why that would be the case.
- Dylan16807 10y ago
  >Do you think written Standard English is less ambiguous than any other English Definitely. Things are clearer when you write formally and strictly adhere to grammatical rules. When I'm speaking casually I omit words and (ab)use punctuation much differently, and while that is perfectly acceptable to do, it is more complex and ambiguous to parse. As far as I'm aware the same pattern exists in a lot of languages and dialects, and it's possible to define formal AAVE too.
  - deleted 10y ago
    [deleted]
  - gugagore 10y ago
    I don't think there is anything "formal" about Standard English. Can you give an example of what you're talking about? Which grammatical rules do you have in mind? I don't know what you mean by formal black English (AAVE) vs. ... casual black English?
  - Dylan16807 10y ago
    To your comment that was detected as a duplicate: >I don't think there is anything "formal" about Standard English. "Standard" English as she is spoke? No. But a3n was specifically talking about the formal version often called "Standard written English". It is very prescriptive and less flexible, which makes it easier to parse. > Can you give an example of what you're talking about? Which grammatical rules do you have in mind? Let's start with "every rule involving punctuation".
    - gugagore 10y ago
      I'm not trying to be dense, but I still don't see it. Punctuation helps with identifying boundaries of clauses, whether they're dependent or not, whether a sentence is a question. Is this what NLP struggles with? I don't think so. One of the big unsolved problems is understanding the referents of pronouns (and other proforms, like "I will [do so] too"). Style guides and generally prescriptivism does not help that. Another ambiguity is parse ambiguities like " I saw the girl with the telescope." What does "formal" English say about that?

deleted 10y ago

[deleted]

TazeTSchnitzel 10y ago

Partly that, but my guess would be that it's because it's largely trained from written English (not the sort you see on Twitter and in SMS), and written English is usually one of the standard varieties, and the standard varieties bear little resemblance to AAVE. I would guess the same problem occurs with other dialects which are sufficiently different from the standard.

viraptor 10y ago

I expect it's mostly a reflection on the learning data in language processing. Almost any corpus of text you can easily find for experimenting is close to standard English. Dialects very rarely appear in published texts unless it's just to point out that a specific person being quoted uses that dialect. Choose a student of any colour. When they need a text corpus for research, do you think they'll reach for a different, known, large text collection that matches their background?

leftpad 10y ago

This is absolutely one of the reasons. See Google's computer vision system classifying black folks as gorillas: http://blogs.wsj.com/digits/2015/07/01/google-mistakenly-tags-black-people-as-gorillas-showing-limits-of-algorithms/ http://blogs.wsj.com/digits/2015/07/01/google-mistakenly-tag...

gjolund 10y ago

Do you know of any AI's being written by predominately black teams?