7 ms·

Machine Bias: Man Is to Computer Programmer as Woman Is to Homemaker?

15 points by acoravos 9y ago

backpropaganda 9y ago
If I were training a classifier to predict whether a sentence is talking about household activities v/s not, wouldn't the occurrence of man/woman in the sentence be a good feature? Today, woman do perform household activities more (whether we like it or not), and wouldn't it make sense to use that piece of information when performing some predictive analysis? The technical sense of "bias" arises when the train and test distributions differ. Obviously if you train with a dataset of text from a foreign country's news and then apply it on an American context, the difference in the data distributions will introduce bias, but why do we need a social twist to this already well-functioning term? If the same classifier is trained and evaluated in India (with its sexist roles, say), then there's no (technical) bias and I don't see why it's a bad application.
- praxulus 9y ago
  >wouldn't it make sense to use that piece of information when performing some predictive analysis? No, because eventually your system will graduate from predicting the results of society's bias to reinforcing society's bias. That is a bad thing.
  - backpropaganda 9y ago
    Can you give an example of a situation where an ML application would be reinforcing a problematic bias but still have good performance metrics? My point is that a wrongly-applied ML application would suffer in just plain accuracy. For instance, a Automatic Carrier Counsellor might give "homemaker" as a suggested career choice to women, but then before we start calling it biased, it would already be wrong. If the same algorithm had dug deeper, it would have learn that the said woman would be a great programmer.
    - praxulus 9y ago
      Recidivism prediction systems will usually tell you that black people are more likely to get arrested/convicted again. They do so accurately, but also result in longer sentences for black people. https://arxiv.org/abs/1610.07524 https://arxiv.org/abs/1610.07524
      - throwanem 9y ago
        That sounds like exactly the kind of thing you'd expect to happen when you treat people as feature clusters instead of, you know, people.
      - xupybd 9y ago
        Yeah but doesn't that have more to do with the way the predictions are used? It seems to me to be a stupid thing to do. This person seems more likely to get convicted again, lock 'em up longer. Instead of asking why is this person more likely to get convicted again? Can we prevent this in a redemptive non punitive way? It's really useful to have that prediction/data but how you use it is more important
        ZenPsycho 9y ago
        the problem is a layperson doesn't necessarily know what a prediction necessarily means without a deep understanding of how the system is making its predictions, let alone how to apply it. worse is that since the prediction is coming from computer that lends the prediction an air of authority another article called "bias laundering". the general belief is that computers are objective and cannot have bias, which in a sense is true, but people don't tend to think a step further about the problems and biases in the people who programmed the computer. so that is definitely a thing usually missing from these discussions is that the people using these systems generally don't know how they work, and believe they predict or imply things that they don't
      - industriousthou 9y ago
        I mean, that same algorithm could be used to determine that blacks or other at-risk groups should receive extra attention or support. An accurate picture of reality can be used poorly or well.
- xupybd 9y ago
  I think you have a really good point here. The problem is that we have this current bias in society and people wish to change it. I think there is a fear, that if we reflect this bias, in the way we talk, we re-enforce the bias. It seems an effective tool, if you want to change thinking then police the way words can be used around the topic. It is however worrying that machines could start playing a role in this. It could become a powerful tool in steering public opinion. This doesn't seem too bad, but that could be used to favour an incumbent political party, or more than likely to sell products we otherwise don't really want. But you are right machines need accuracy and removing that bias could be detrimental to the task they're solving.
  - backpropaganda 9y ago
    My point isn't that accuracy and bias are orthogonal, but that bias is contained in the accuracy metric.
    - xupybd 9y ago
      Totally agree with you. I'm not at all trying to say accuracy and bias can be orthogonal. I'm trying to say some people think they have a good enough reason to throw away accuracy if that means they can change a societal bias. But that can only be a good thing if you agree with the change being made.
    - anigbrowl 9y ago
      That's fine if you are measuring the bias component too. If you're not, you risk perpetuating it. It's natural if I read a sentence about hands to conclude that the test refers to people and not fish, but if I read about 'the hands of a surgeon' and assume that those hands are male based on the current demographics of the surgical profession, I'm making an unwarranted assumption on insufficient information. It's wise to grant utility to uncertainty by maintaining a 'Don't know' option rather than being in a rush to make a determination before it is necessary, not least because of the computational cost of unwinding incorrect assumptions.
    - industriousthou 9y ago
- deleted 9y ago
  [deleted]
- anigbrowl 9y ago
  No, it would not be a good feature. For one thing, baking the bias of existing practices as opposed to constraints risks reinforcing that practice as more and more decision-making is left to ML. Second it makes your system vulnerable to verbal paradoxes designed to exploit that bias.
reader5000 9y ago
In the sjw-religion, why is "homemaker" considered inferior to "computer programmer"? One of the oldest and most important human occupations versus hunched over at a desk slaving for a salary until being outsourced to a bot in 5 years? I've never understood the default sjw/"feminism" assumptions that anything feminine is "bad".
- angmarsbane 9y ago
  It isn't so much bad or negative as it is risky. A homemaker, male or female, becomes financially reliant on his/her partner. A partner who can die, become too disabled/ill to work, or who can leave after the homemaker has missed his/her key career/skill building years. Women are more likely to be pushed or encouraged to take this important supportive role to benefit others while putting themselves at risk.
  - AstralStorm 9y ago
    It goes both ways, the partner can become reliant on this kind of support. And only sometimes partly you can cover the difference with money and/or power - typically much more expensive than cooperation. In fact I'd say the housekeeping skills are always more marketable, more basic and easier to master - you can live off them, but not without. This gives rise to competition which drives both social perception of value and actual financial value down. Mutual dependency has been the way of life for ages - for good reason. Edit: Thought police strikes again! Instead of downvoting, please provide a coherent argument why a given point is invalid or how.
    - angmarsbane 9y ago
      It is a lot easier to hire someone to take care of housekeeping duties or to do them yourself than it is to make up for 10+ years of little to zero non-homemaking skill development.
      - AstralStorm 9y ago
        Not as easy as you'd expect if you want someone actually competent and versatile. You would be surprised how big a bag "homemaking" is, ranging from cooking, through teaching, down to clerical work, through basic finances and back up with handyman (yes yes) fixes. Any of those skills specialized in is marketable, though not respectable on its own. Note how few of them are true knowledge and research work. All of them are in the areas where there have been major reductions in number of jobs due to automation and centralization.
vtange 9y ago
This is the tug-o-war of influencer v. influencee. A machine that just tells-it-as-it-is might hold an advantage over one that willingly ignores some data to promote a different view of the world. Personally, I see more danger in people trying to make machines that evangelize their own biases to the world than machines being molded by the existing social assumptions of society, given that we expect machines to perform most of the work/control most of the resources in the future.
mkrum 9y ago
If you are going to "debias" your model, what is the point of even training the model to handle these issues in the first place? Not surprisingly, human language can be biased. If you train a model on human language it will not magically transcend those biases. The problem is that people have this expectation that ML is going to lead to these perfect decision makers. Machine Learning creates models that reflect the data, not the truth.

15 points by acoravos 9y ago

If I were training a classifier to predict whether a sentence is talking about household activities v/s not, wouldn't the occurrence of man/woman in the sentence be a good feature? Today, woman do perform household activities more (whether we like it or not), and wouldn't it make sense to use that piece of information when performing some predictive analysis? The technical sense of "bias" arises when the train and test distributions differ. Obviously if you train with a dataset of text from a foreign country's news and then apply it on an American context, the difference in the data distributions will introduce bias, but why do we need a social twist to this already well-functioning term? If the same classifier is trained and evaluated in India (with its sexist roles, say), then there's no (technical) bias and I don't see why it's a bad application.

praxulus 9y ago
>wouldn't it make sense to use that piece of information when performing some predictive analysis? No, because eventually your system will graduate from predicting the results of society's bias to reinforcing society's bias. That is a bad thing.
- backpropaganda 9y ago
  Can you give an example of a situation where an ML application would be reinforcing a problematic bias but still have good performance metrics? My point is that a wrongly-applied ML application would suffer in just plain accuracy. For instance, a Automatic Carrier Counsellor might give "homemaker" as a suggested career choice to women, but then before we start calling it biased, it would already be wrong. If the same algorithm had dug deeper, it would have learn that the said woman would be a great programmer.
  - praxulus 9y ago
    Recidivism prediction systems will usually tell you that black people are more likely to get arrested/convicted again. They do so accurately, but also result in longer sentences for black people. https://arxiv.org/abs/1610.07524 https://arxiv.org/abs/1610.07524
    - throwanem 9y ago
      That sounds like exactly the kind of thing you'd expect to happen when you treat people as feature clusters instead of, you know, people.
    - xupybd 9y ago
      Yeah but doesn't that have more to do with the way the predictions are used? It seems to me to be a stupid thing to do. This person seems more likely to get convicted again, lock 'em up longer. Instead of asking why is this person more likely to get convicted again? Can we prevent this in a redemptive non punitive way? It's really useful to have that prediction/data but how you use it is more important
      - ZenPsycho 9y ago
        the problem is a layperson doesn't necessarily know what a prediction necessarily means without a deep understanding of how the system is making its predictions, let alone how to apply it. worse is that since the prediction is coming from computer that lends the prediction an air of authority another article called "bias laundering". the general belief is that computers are objective and cannot have bias, which in a sense is true, but people don't tend to think a step further about the problems and biases in the people who programmed the computer. so that is definitely a thing usually missing from these discussions is that the people using these systems generally don't know how they work, and believe they predict or imply things that they don't
    - industriousthou 9y ago
      I mean, that same algorithm could be used to determine that blacks or other at-risk groups should receive extra attention or support. An accurate picture of reality can be used poorly or well.
xupybd 9y ago
I think you have a really good point here. The problem is that we have this current bias in society and people wish to change it. I think there is a fear, that if we reflect this bias, in the way we talk, we re-enforce the bias. It seems an effective tool, if you want to change thinking then police the way words can be used around the topic. It is however worrying that machines could start playing a role in this. It could become a powerful tool in steering public opinion. This doesn't seem too bad, but that could be used to favour an incumbent political party, or more than likely to sell products we otherwise don't really want. But you are right machines need accuracy and removing that bias could be detrimental to the task they're solving.
- backpropaganda 9y ago
  My point isn't that accuracy and bias are orthogonal, but that bias is contained in the accuracy metric.
  - xupybd 9y ago
    Totally agree with you. I'm not at all trying to say accuracy and bias can be orthogonal. I'm trying to say some people think they have a good enough reason to throw away accuracy if that means they can change a societal bias. But that can only be a good thing if you agree with the change being made.
  - anigbrowl 9y ago
    That's fine if you are measuring the bias component too. If you're not, you risk perpetuating it. It's natural if I read a sentence about hands to conclude that the test refers to people and not fish, but if I read about 'the hands of a surgeon' and assume that those hands are male based on the current demographics of the surgical profession, I'm making an unwarranted assumption on insufficient information. It's wise to grant utility to uncertainty by maintaining a 'Don't know' option rather than being in a rush to make a determination before it is necessary, not least because of the computational cost of unwinding incorrect assumptions.
  - industriousthou 9y ago
deleted 9y ago
[deleted]
anigbrowl 9y ago
No, it would not be a good feature. For one thing, baking the bias of existing practices as opposed to constraints risks reinforcing that practice as more and more decision-making is left to ML. Second it makes your system vulnerable to verbal paradoxes designed to exploit that bias.

reader5000 9y ago

In the sjw-religion, why is "homemaker" considered inferior to "computer programmer"? One of the oldest and most important human occupations versus hunched over at a desk slaving for a salary until being outsourced to a bot in 5 years? I've never understood the default sjw/"feminism" assumptions that anything feminine is "bad".

angmarsbane 9y ago
It isn't so much bad or negative as it is risky. A homemaker, male or female, becomes financially reliant on his/her partner. A partner who can die, become too disabled/ill to work, or who can leave after the homemaker has missed his/her key career/skill building years. Women are more likely to be pushed or encouraged to take this important supportive role to benefit others while putting themselves at risk.
- AstralStorm 9y ago
  It goes both ways, the partner can become reliant on this kind of support. And only sometimes partly you can cover the difference with money and/or power - typically much more expensive than cooperation. In fact I'd say the housekeeping skills are always more marketable, more basic and easier to master - you can live off them, but not without. This gives rise to competition which drives both social perception of value and actual financial value down. Mutual dependency has been the way of life for ages - for good reason. Edit: Thought police strikes again! Instead of downvoting, please provide a coherent argument why a given point is invalid or how.
  - angmarsbane 9y ago
    It is a lot easier to hire someone to take care of housekeeping duties or to do them yourself than it is to make up for 10+ years of little to zero non-homemaking skill development.
    - AstralStorm 9y ago
      Not as easy as you'd expect if you want someone actually competent and versatile. You would be surprised how big a bag "homemaking" is, ranging from cooking, through teaching, down to clerical work, through basic finances and back up with handyman (yes yes) fixes. Any of those skills specialized in is marketable, though not respectable on its own. Note how few of them are true knowledge and research work. All of them are in the areas where there have been major reductions in number of jobs due to automation and centralization.

vtange 9y ago

This is the tug-o-war of influencer v. influencee. A machine that just tells-it-as-it-is might hold an advantage over one that willingly ignores some data to promote a different view of the world. Personally, I see more danger in people trying to make machines that evangelize their own biases to the world than machines being molded by the existing social assumptions of society, given that we expect machines to perform most of the work/control most of the resources in the future.

mkrum 9y ago

If you are going to "debias" your model, what is the point of even training the model to handle these issues in the first place? Not surprisingly, human language can be biased. If you train a model on human language it will not magically transcend those biases. The problem is that people have this expectation that ML is going to lead to these perfect decision makers. Machine Learning creates models that reflect the data, not the truth.