4 ms·

To add to the other answers, machine learning based on biased datasets results in biased models. For example, Word2Vec can be used to make analogies. Having i

To add to the other answers, machine learning based on biased datasets results in biased models. For example, Word2Vec can be used to make analogies. Having it determine "Man is to woman as king is to ___." results in "queen", which is reasonable. On the other hand, determining "Man is to computer programmer as woman is to ___." results in "homemaker". The algorithm wasn't deliberately designed to have sexist bias in it, but there was implicit bias in the dataset from which it learned. https://www.technologyreview.com/2016/07/27/158634/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/ https://www.technologyreview.com/2016/07/27/158634/how-vecto... As another example, predictive policing tries to place police in places with higher crime rates. Those crime rates are determined by looking at past history of police reports and arrests. That past history has human bias already in it, with disproportionately higher arrest rates in places with racial minorities. The effect of the predictive policing is to justify overpolicing of minorities. https://en.wikipedia.org/wiki/Predictive_policing#Criticisms https://en.wikipedia.org/wiki/Predictive_policing#Criticisms

mattkrause 6y ago
The analogy thing didn’t entirely hold up: the original demonstration was constrained not to return the same word in the prompt, so “man is to woman as doctor is to ____” had to return something that’s close to, but not the same as “doctor” in the embedding space. Hence, it returns nurse. Ironically, this makes the original point nearly as well: we need to evaluate the hell out of machine learning systems to make sure that they’re doing what we think they are and that they’re not keying off something else instead, especially something biased. To date, the field has been...not great about this.
- organsnyder 6y ago
  I wonder what it would return for "Woman is to doctor as man is to ___". I'd bet money on the result not being "nurse".
  - sqrt17 6y ago
    It highly depends on the corpus and you can check for yourself. I'm being lazy, but the results for Gutenberg books you can check online at http://labs.statsbiblioteket.dk/dsc/ http://labs.statsbiblioteket.dk/dsc/ - man is to woman as doctor is to reprovingly (nurse is the first noun, on position 4) - woman is to man as doctor is to snodgrass (after a couple nonsense/rare words) The most important thing that teaches us is that big corpora (bigger than PG) are essential for this method.
  - mattkrause 6y ago
    You can monkey around with it here: http://bionlp-www.utu.fi/wv_demo/ http://bionlp-www.utu.fi/wv_demo/ (choose the English Google News model, but this may not be exactly the same set/model as the original report). Man is to Woman as Doctor is to ___ gives 1) gynecologist 2) nurse 3) doctors 4) physician 5) pediatrician Woman is to Man as Doctor is to ___ gives: 1) physician 2) doctors 3) surgeon 4) dentist 5) cardiologist These are just generally near "Doctor" though: the ten nearest terms are physician, doctors, gynecologist, surgeon, dentist, pediatrician, pharmacist, neurologist, cardiologist, and nurse. Some gender differences may persist (nurse is #2 for `woman`, but #68 for `man`, but it's also near `woman` generally and you could imagine it gets a bit of a boost from the verb ("to feed a baby") being attached exclusively to women too. Anyway, my point is not that there's no bias (there certainly can be--seed GTP-3 with a prompt about Muslims) but that one should be wary of thinking they know what the model is doing.
bigyikes 6y ago
How would you expect the computer programmer analogy be completed? I legitimately don’t know what a reasonable answer would be.
- mulmen 6y ago
  The takeaway is the model has bias. It’s up to humans to decide what to do with that information. Depending on the input “homemaker” may be a technically reasonable output. Should we use such a model to suggest career paths to high school students? Or should we reevaluate the methodology?
  - lopmotr 6y ago
    Who cares about ML when we already do that: https://www.slideshare.net/yuyomajadero/jobs-occupations-professions-pictionary-poster-vocabulary-worksheet-1 https://www.slideshare.net/yuyomajadero/jobs-occupations-pro...
    - deleted 6y ago
      [deleted]
    - mulmen 6y ago
      The danger is in thinking ML can magically see past human biases and/or is not subject to it’s own types of bias.
      - lopmotr 6y ago
        Seems like an overhyped danger. Does anyone actually think that? You would have to not even know that the purpose of a language model is to model human language. But if that's the level of your understanding, why would you select a language model to use as a career guidance oracle? Such a negligent career advisor would be just as likely to use those children's posters with pictures of female nurses and male doctors. I'd like to add that there's also a danger in people trying too hard to avoid bias and losing important information. For example, a man who's a homemaker instead of a programmer is a less attractive partner for a woman. So such an occupation might harm both his quality of life and that of his partner. There is some useful information encoded in cultural bias. Even if that information turns out to be entirely socially constructed, it still has real harmful effects on real humans who go against it.
visarga 6y ago
> On the other hand, determining "Man is to computer programmer as woman is to ___." results in "homemaker" Who are we to decide for the woman what she should feel about it? Is bias absolute or relative, objective or subjective? Is there a one true policy for dealing with bias or can there be one?

mattkrause 6y ago

The analogy thing didn’t entirely hold up: the original demonstration was constrained not to return the same word in the prompt, so “man is to woman as doctor is to ____” had to return something that’s close to, but not the same as “doctor” in the embedding space. Hence, it returns nurse. Ironically, this makes the original point nearly as well: we need to evaluate the hell out of machine learning systems to make sure that they’re doing what we think they are and that they’re not keying off something else instead, especially something biased. To date, the field has been...not great about this.

organsnyder 6y ago
I wonder what it would return for "Woman is to doctor as man is to ___". I'd bet money on the result not being "nurse".
- sqrt17 6y ago
  It highly depends on the corpus and you can check for yourself. I'm being lazy, but the results for Gutenberg books you can check online at http://labs.statsbiblioteket.dk/dsc/ http://labs.statsbiblioteket.dk/dsc/ - man is to woman as doctor is to reprovingly (nurse is the first noun, on position 4) - woman is to man as doctor is to snodgrass (after a couple nonsense/rare words) The most important thing that teaches us is that big corpora (bigger than PG) are essential for this method.
- mattkrause 6y ago
  You can monkey around with it here: http://bionlp-www.utu.fi/wv_demo/ http://bionlp-www.utu.fi/wv_demo/ (choose the English Google News model, but this may not be exactly the same set/model as the original report). Man is to Woman as Doctor is to ___ gives 1) gynecologist 2) nurse 3) doctors 4) physician 5) pediatrician Woman is to Man as Doctor is to ___ gives: 1) physician 2) doctors 3) surgeon 4) dentist 5) cardiologist These are just generally near "Doctor" though: the ten nearest terms are physician, doctors, gynecologist, surgeon, dentist, pediatrician, pharmacist, neurologist, cardiologist, and nurse. Some gender differences may persist (nurse is #2 for `woman`, but #68 for `man`, but it's also near `woman` generally and you could imagine it gets a bit of a boost from the verb ("to feed a baby") being attached exclusively to women too. Anyway, my point is not that there's no bias (there certainly can be--seed GTP-3 with a prompt about Muslims) but that one should be wary of thinking they know what the model is doing.

bigyikes 6y ago

How would you expect the computer programmer analogy be completed? I legitimately don’t know what a reasonable answer would be.

mulmen 6y ago
The takeaway is the model has bias. It’s up to humans to decide what to do with that information. Depending on the input “homemaker” may be a technically reasonable output. Should we use such a model to suggest career paths to high school students? Or should we reevaluate the methodology?
- lopmotr 6y ago
  Who cares about ML when we already do that: https://www.slideshare.net/yuyomajadero/jobs-occupations-professions-pictionary-poster-vocabulary-worksheet-1 https://www.slideshare.net/yuyomajadero/jobs-occupations-pro...
  - deleted 6y ago
    [deleted]
  - mulmen 6y ago
    The danger is in thinking ML can magically see past human biases and/or is not subject to it’s own types of bias.
    - lopmotr 6y ago
      Seems like an overhyped danger. Does anyone actually think that? You would have to not even know that the purpose of a language model is to model human language. But if that's the level of your understanding, why would you select a language model to use as a career guidance oracle? Such a negligent career advisor would be just as likely to use those children's posters with pictures of female nurses and male doctors. I'd like to add that there's also a danger in people trying too hard to avoid bias and losing important information. For example, a man who's a homemaker instead of a programmer is a less attractive partner for a woman. So such an occupation might harm both his quality of life and that of his partner. There is some useful information encoded in cultural bias. Even if that information turns out to be entirely socially constructed, it still has real harmful effects on real humans who go against it.

visarga 6y ago

> On the other hand, determining "Man is to computer programmer as woman is to ___." results in "homemaker" Who are we to decide for the woman what she should feel about it? Is bias absolute or relative, objective or subjective? Is there a one true policy for dealing with bias or can there be one?