5 ms·

This is an interesting ethical challenge. If the training corpus for a machine learning model contains stereotypes and biases, then the output of the model wil

by panarky 10y ago

This is an interesting ethical challenge. If the training corpus for a machine learning model contains stereotypes and biases, then the output of the model will reflect those prejudices. When that model is used in large-scale applications, it will not just repeat those biases, it will amplify them. It propagates biased language, which feeds back into the model in a self-reinforcing vicious cycle of increasing bias. Should we attempt to detect prejudices in machine learning models and actively counter them, creating a self-reinforcing virtuous cycle of balance and tolerance? ---- As an aside, for an article about language, this one has a lot of typos. Doesn't anyone proofread before posting?

bmmayer1 10y ago
Especially as it relates to this: "This means that blogs or websites that employ African-American language could actually be pushed down in search results because of Google’s language processing." Isn't the ideal system to not assume anything about the content it's analyzing, but to base the model on behavioral patterns instead? In the case of Google ranking, either a site has traffic/low bounce/backlinks/social cred or it doesn't--why is Google attempting to 'read' and analyze the content itself? (Other than to serve ads of course)
- deleted 10y ago
  [deleted]
- panarky 10y ago
  > Isn't the ideal system to not assume anything about the content it's analyzing ... Because Google no longer just searches for literal strings of text, ranked by links. They now infer your meaning and find results that match your intent -- even if the exact search string doesn't exist in the search results. For more on this, read about Hinton's work on "thought vectors" [1]. So if Google can't determine the meaning of African American language as well as it can standard English, then that content is likely to be less visible in search results. [1] https://www.theguardian.com/science/2015/may/21/google-a-step-closer-to-developing-machines-with-human-like-intelligence https://www.theguardian.com/science/2015/may/21/google-a-ste...
  - ams6110 10y ago
    I can't say I would think that Google does a particularly good job at interpreting the intent of my search queries. I often need to reformulate them to get what I'm after. But I doubt that my searches are similar to those of the general public so that is understandable.
    - visarga 10y ago
      I get the same feeling - that Google is often missing the meaning and giving me unrelated keyword matches.
      - alphonsegaston 10y ago
        It's even worse than that - it explicitly ignores quoted keywords in many cases. The only intent they care about is what guides you towards their interests.
        rasz_pl 10y ago
        my favorite is from today when searching for "comparison prohibited benchmarks" google decide to give me results to "comparison ALLOWED benchmarks"
  - protomyth 10y ago
    "They now infer your meaning and find results that match your intent -- even if the exact search string doesn't exist in the search results." Bing seems to do the same thing. I really wish there were a grep for the web because both engines get it wrong even when I try to exclude results.
    - Tiksi 10y ago
      Try verbatim mode with google: http://paste.click/SWqasi http://paste.click/SWqasi It's basically grep for the web, since it uses the actual query instead of the meaning of it.
- TazeTSchnitzel 10y ago
  Showing only (or prioritising) same-language results helps with relevancy. A German speaker who searches for “gift” is not looking for ideas for presents, and an Italian speaker who searches for “peperoni” is not looking for the American variant of salami. A further problem, though, is that search engines try to understand the query and the content searched at a deeper level than mere keyword matches. For example, they may try to pick up synonyms (if I search for “movies in <local area>”, perhaps results for “films” might be helpful too) and associations.
- yongjik 10y ago
  > why is Google attempting to 'read' and analyze the content itself? So that when you search for "cheesecake recipes", you don't get a well-sourced, high-traffic article from a respectable website describing in detail how the new business initiative by Cheesecake Factory is a recipe for disaster. * Disclaimer: I pulled that example out of thin air. There's actually no such article.
zxcvvcxz 10y ago
> When that model is used in large-scale applications, it will not just repeat those biases, it will amplify them. It propagates biased language, which feeds back into the model in a self-reinforcing vicious cycle of increasing bias. Which is precisely why social media (e.g. your newsfeed) is such an awful source for news, especially when you consider how low the standards are on digital publishing these days...
a3n 10y ago
> Doesn't anyone proofread before posting? No.
bsbechtel 10y ago
So humans do this too naturally - this is how new languages develop over hundreds of years and groups evolve and split. Aside from that, I'm a little nervous about seeing machines accelerate this process.
wyager 10y ago
> If the training corpus for a machine learning model contains stereotypes and biases, then the output of the model will reflect those prejudices. Depends what you mean by "the corpus". Let's take the example of loan applications. Maybe the training data contains credit scores, biographical information, and personal essays about the applicants from loan officers. Let's say loan officers tend to use more negative language when talking about black people. Any decent machine learning system given racial data will not adopt those biases. In fact, it will counteract them. It will determine that e.g. the semantic content of the essays is an indicator of loan suitability, and it will also notice if it has to e.g. add something to the essay score for black people to make the best decisions. As long as the output data is objective, competent ML systems will account for any bias in the input data. > Should we attempt to detect prejudices in machine learning models We don't have to. If the output examples we are training on are true, the ML algorithm won't adopt any incorrect biases. This is literally the point of machine learning. You want to make the best possible decision. If your algorithm has incorrect human biases, you're losing money. ML naturally accounts for this stuff.
- gugagore 10y ago
  You are only talking about supervised learning. And the bias people are probably talking about is bias in the data, not bias in the "input", as you put it. See http://arxiv.org/abs/1607.06520 http://arxiv.org/abs/1607.06520 for a counterexample to your point.
  - wyager 10y ago
    In that paper it sounds like the algorithm is finding true relationships that the author doesn't like. What's the problem with the algorithm?
    - gugagore 10y ago
      I don't understand what you mean by "the algorithm" or "true relationships". Do you know what I mean? There's truth as in what the data says, and there's truth as in what's underlying before social processes " corrupt " the truth. This paper examines ways to get at the latent truth. Do you think that's a useful endeavor? It's dismissive to say it has anything to do with what the "author likes".
gugagore 10y ago
Yes! Here's an example: http://arxiv.org/abs/1607.06520 http://arxiv.org/abs/1607.06520
alphonsegaston 10y ago
Since most languages have dialects of similar "standard" form divergence, it's an NLP problem that would have to be addressed anyway. While slang might be hard to keep up with, the grammatical patterns of AAVE are well-documented and understood by linguists, and there's plenty of data to model. The issue of prejudice stems from refusing to treat AAVE as a valid input to an AI, mirroring the racism in American culture that similarly devalues and excludes it. Output can conform to some approximation of "standard" form without being problematic, following the pattern of interaction between standard and dialect speakers. As a speaker of the standard, it's gauche and derisive to imitate or refuse or comprehend a dialect speaker, not to respond in your own form.
WhoBeI 10y ago
Was reading the article and, in true wiki fashion, click the links that seem relevant to get more context. As is often the case I ended up in some other corner of the internet watching interviews with Stephen Pinker. Something in all that input triggered a thought when I read your comment: What is really the difference between spelling a word wrongly and using a non-dictionary word to express something. We except one as "slang" but will criticize the other because they don't comply with established rules. Isn't that what slang is? A break from established rules. Anyways, just a thought, because I agree with you; if you publish - proofread. I just find it interesting to throw a Socrates argument at myself from time to time. Btw and somewhat of topic.. this was the Pinker interview that was most interesting: https://www.youtube.com/watch?v=egU0dxzFKAQ https://www.youtube.com/watch?v=egU0dxzFKAQ
sevenless 10y ago
This is the same question as 'What values should we impose on society?' And that is an irreducible conflict. It's politics. Programming a computer to perform a task is a powerful way to reveal hidden assumptions. What the problem of machine learning on human data reveals, is there is no objectivity. Our most 'objective' models will simply learn and then reinforce biases and prejudices and cause harm to people. We can't build objective, neutral, value-free models when it comes to human behavior, because humans change their behavior in response to the models. When we reject stereotypes we are imposing a set of values, just as much as when we embrace them. Machine learning forces us to confront the fact that this is exactly what we are doing. There's a kind of philosophical crisis going on here. We need an entirely new way to think about the limits of objectivity in human sciences, and how to create ethical models in the presence of feedback between science and society. The language of objective physical science doesn't work.

by panarky 10y ago

bmmayer1 10y ago

Especially as it relates to this: "This means that blogs or websites that employ African-American language could actually be pushed down in search results because of Google’s language processing." Isn't the ideal system to not assume anything about the content it's analyzing, but to base the model on behavioral patterns instead? In the case of Google ranking, either a site has traffic/low bounce/backlinks/social cred or it doesn't--why is Google attempting to 'read' and analyze the content itself? (Other than to serve ads of course)

deleted 10y ago
[deleted]
panarky 10y ago
> Isn't the ideal system to not assume anything about the content it's analyzing ... Because Google no longer just searches for literal strings of text, ranked by links. They now infer your meaning and find results that match your intent -- even if the exact search string doesn't exist in the search results. For more on this, read about Hinton's work on "thought vectors" [1]. So if Google can't determine the meaning of African American language as well as it can standard English, then that content is likely to be less visible in search results. [1] https://www.theguardian.com/science/2015/may/21/google-a-step-closer-to-developing-machines-with-human-like-intelligence https://www.theguardian.com/science/2015/may/21/google-a-ste...
- ams6110 10y ago
  I can't say I would think that Google does a particularly good job at interpreting the intent of my search queries. I often need to reformulate them to get what I'm after. But I doubt that my searches are similar to those of the general public so that is understandable.
  - visarga 10y ago
    I get the same feeling - that Google is often missing the meaning and giving me unrelated keyword matches.
    - alphonsegaston 10y ago
      It's even worse than that - it explicitly ignores quoted keywords in many cases. The only intent they care about is what guides you towards their interests.
      - rasz_pl 10y ago
        my favorite is from today when searching for "comparison prohibited benchmarks" google decide to give me results to "comparison ALLOWED benchmarks"
- protomyth 10y ago
  "They now infer your meaning and find results that match your intent -- even if the exact search string doesn't exist in the search results." Bing seems to do the same thing. I really wish there were a grep for the web because both engines get it wrong even when I try to exclude results.
  - Tiksi 10y ago
    Try verbatim mode with google: http://paste.click/SWqasi http://paste.click/SWqasi It's basically grep for the web, since it uses the actual query instead of the meaning of it.
TazeTSchnitzel 10y ago
Showing only (or prioritising) same-language results helps with relevancy. A German speaker who searches for “gift” is not looking for ideas for presents, and an Italian speaker who searches for “peperoni” is not looking for the American variant of salami. A further problem, though, is that search engines try to understand the query and the content searched at a deeper level than mere keyword matches. For example, they may try to pick up synonyms (if I search for “movies in <local area>”, perhaps results for “films” might be helpful too) and associations.
yongjik 10y ago
> why is Google attempting to 'read' and analyze the content itself? So that when you search for "cheesecake recipes", you don't get a well-sourced, high-traffic article from a respectable website describing in detail how the new business initiative by Cheesecake Factory is a recipe for disaster. * Disclaimer: I pulled that example out of thin air. There's actually no such article.

zxcvvcxz 10y ago

> When that model is used in large-scale applications, it will not just repeat those biases, it will amplify them. It propagates biased language, which feeds back into the model in a self-reinforcing vicious cycle of increasing bias. Which is precisely why social media (e.g. your newsfeed) is such an awful source for news, especially when you consider how low the standards are on digital publishing these days...

a3n 10y ago

> Doesn't anyone proofread before posting? No.

bsbechtel 10y ago

So humans do this too naturally - this is how new languages develop over hundreds of years and groups evolve and split. Aside from that, I'm a little nervous about seeing machines accelerate this process.

wyager 10y ago

> If the training corpus for a machine learning model contains stereotypes and biases, then the output of the model will reflect those prejudices. Depends what you mean by "the corpus". Let's take the example of loan applications. Maybe the training data contains credit scores, biographical information, and personal essays about the applicants from loan officers. Let's say loan officers tend to use more negative language when talking about black people. Any decent machine learning system given racial data will not adopt those biases. In fact, it will counteract them. It will determine that e.g. the semantic content of the essays is an indicator of loan suitability, and it will also notice if it has to e.g. add something to the essay score for black people to make the best decisions. As long as the output data is objective, competent ML systems will account for any bias in the input data. > Should we attempt to detect prejudices in machine learning models We don't have to. If the output examples we are training on are true, the ML algorithm won't adopt any incorrect biases. This is literally the point of machine learning. You want to make the best possible decision. If your algorithm has incorrect human biases, you're losing money. ML naturally accounts for this stuff.

gugagore 10y ago
You are only talking about supervised learning. And the bias people are probably talking about is bias in the data, not bias in the "input", as you put it. See http://arxiv.org/abs/1607.06520 http://arxiv.org/abs/1607.06520 for a counterexample to your point.
- wyager 10y ago
  In that paper it sounds like the algorithm is finding true relationships that the author doesn't like. What's the problem with the algorithm?
  - gugagore 10y ago
    I don't understand what you mean by "the algorithm" or "true relationships". Do you know what I mean? There's truth as in what the data says, and there's truth as in what's underlying before social processes " corrupt " the truth. This paper examines ways to get at the latent truth. Do you think that's a useful endeavor? It's dismissive to say it has anything to do with what the "author likes".

gugagore 10y ago

Yes! Here's an example: http://arxiv.org/abs/1607.06520 http://arxiv.org/abs/1607.06520

alphonsegaston 10y ago

Since most languages have dialects of similar "standard" form divergence, it's an NLP problem that would have to be addressed anyway. While slang might be hard to keep up with, the grammatical patterns of AAVE are well-documented and understood by linguists, and there's plenty of data to model. The issue of prejudice stems from refusing to treat AAVE as a valid input to an AI, mirroring the racism in American culture that similarly devalues and excludes it. Output can conform to some approximation of "standard" form without being problematic, following the pattern of interaction between standard and dialect speakers. As a speaker of the standard, it's gauche and derisive to imitate or refuse or comprehend a dialect speaker, not to respond in your own form.

WhoBeI 10y ago

Was reading the article and, in true wiki fashion, click the links that seem relevant to get more context. As is often the case I ended up in some other corner of the internet watching interviews with Stephen Pinker. Something in all that input triggered a thought when I read your comment: What is really the difference between spelling a word wrongly and using a non-dictionary word to express something. We except one as "slang" but will criticize the other because they don't comply with established rules. Isn't that what slang is? A break from established rules. Anyways, just a thought, because I agree with you; if you publish - proofread. I just find it interesting to throw a Socrates argument at myself from time to time. Btw and somewhat of topic.. this was the Pinker interview that was most interesting: https://www.youtube.com/watch?v=egU0dxzFKAQ https://www.youtube.com/watch?v=egU0dxzFKAQ

sevenless 10y ago

This is the same question as 'What values should we impose on society?' And that is an irreducible conflict. It's politics. Programming a computer to perform a task is a powerful way to reveal hidden assumptions. What the problem of machine learning on human data reveals, is there is no objectivity. Our most 'objective' models will simply learn and then reinforce biases and prejudices and cause harm to people. We can't build objective, neutral, value-free models when it comes to human behavior, because humans change their behavior in response to the models. When we reject stereotypes we are imposing a set of values, just as much as when we embrace them. Machine learning forces us to confront the fact that this is exactly what we are doing. There's a kind of philosophical crisis going on here. We need an entirely new way to think about the limits of objectivity in human sciences, and how to create ethical models in the presence of feedback between science and society. The language of objective physical science doesn't work.