8 ms·

> "So would any reasonable human." That's the entire point of the article, our biases, as reflected in the world, are learned by models. Consider the following

by gbrown 7y ago

> "So would any reasonable human." That's the entire point of the article, our biases, as reflected in the world, are learned by models. Consider the following: * More computer programmers are men than women (descriptive statement, no problem) * A predictive model correctly identifies that more computer programmers are men than women (a prediction based on observed data, no problem) * A recruiting agency uses a predictive model to recruit computer programmers. Due to the way the model was trained, it excludes qualified women (not OK, a clear misapplication of an algorithm) Feel free to try this thought experiment in other contexts where existing biases can be amplified through algorithms.

swebs 7y ago
I'm fail to see how step 2 leads to step 3. The fact that there are less woman programmers overall should not have any affect on P(good|woman,programmer).
- crooked-v 7y ago
  2 leads to 3 because in a naively designed system, being a man is a stronger predictor of being a good programmer than being a woman, because there are more men than women in high-ranking and high-paying programmer positions.
  - deleted 7y ago
    [deleted]
  - reroute1 7y ago
    Yes but only in very poorly designed systems. We have those now with AI it's no different. I still see no reason to jump from 2 to 3 other than incompetence in design.
    - loopz 7y ago
      The other poster stated "predictive model". A predictive model could easily identify males to be "more fit" (higher p-value) than females, in aggregate. The programmer tries to evade this bias, by ie. removing gender values in the system. However, the system STILL identify males "more fit", due to correllated values (likes Mancester United/Liverpool). Machine Learning is not much more than advanced statistics, so data biases gets amplified, thus naive application is often a poor fit. Even worse, the experts are at loss on how to make such predictive systems "perform better", without introducing biases on their own, since the data and categorizations themselves may be full of biases. In the real world, incompetence may not be a huge hurdle when selling complex systems. Also, these biases are invisible, until one thinks about them or spots them in the wild.
    - crooked-v 7y ago
      Lots of people think "AI" is a magic bullet that will obviate any need to actually think about this stuff.
  - whatshisface 7y ago
    >a stronger predictor of being a good programmer than being a woman Of being a programmer, not being a good programmer. There are more men than women in programming, but the recruiters want to tell good programmers from bad ones, not programmers from nurses. The data clearly shows that a randomly chosen man is more likely to be a programmer than a randomly chosen woman, but that's irrelevant. The likelihood that a randomly chosen female programmer is good should be about the same as the likelihood that a randomly chosen male programmer is good, and that's what hiring managers care about.
    - crooked-v 7y ago
      And yet this kind of problem is happening over and over and over again in real life, such as the automated risk assessments used in Florida deciding that black people are inherently more likely to be criminal than white people. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm...
- ska 7y ago
  That's a central topic of the article linked, that in practice people are finding it difficult to design and train systems without having this effect.
- bluGill 7y ago
  If you are randomly selecting people from the general population it follows. That is if you just grab the basic info of unemployed and call some in for a programming interview culling females from the list before scheduling the interview appointment will save you a lot of time before you find/hire a good programmer. I can't figure out why you would use such an algorithm though. Most people who want to hire programmers look at their resumes from people who have self-selected as programmers (possibly on a job search forum) and interview them. If you decide there are not enough programmers you might hire people at random ("would you be willing to learn to be a programmer if paid?" "yes" "hired, you start Monday") - there might be factors to exclude some people because they can never be great programmers, but I'm not aware of what such factors might be - gender doesn't seem to be one though.
- ncallaway 7y ago
  > I'm fail to see how step 2 leads to step 3. The fact that there are less woman programmers overall should not have any affect on P(good|woman,programmer). You're correct that there should not be any affect. The problem is that many systems are designed such that there is an affect.
  - theamk 7y ago
    Are there such systems? Every paper I saw use trivial models which are unlikely to be used in real life. Like this article, which checks gender associations for words like “money” - that’s not how one would be using AI for hiring in any practical way!
Excel_Wizard 7y ago
Let's say that there is a population bell curve of the variable "propensity to enjoy being a programmer". One curve for men, and one curve for women. In some dystopian future, everyone takes a career aptitude test, and if an individual falls within the top 10% of the overall population, they get turned into a programmer by the government. This is assuming a perfectly-trained AI judges the variable, and that it controls against the underlying variables of societal bias in order to ensure that long-term human resources are properly allocated, and that long-term biases, informed by people's interactions with eachother, trend toward actual biological differences. It may well be the case that the gender split is 80% / 20% male to female (as is roughly the case today). It may not. However, the left-leaning zeitgeist opinion would seem to be that this outcome is impossible, and current observed differences are only due to systematic oppression. The right-leaning zeitgeist opinion would seem to be that this outcome would make sense. I tend to think that the left-leaning opinion on this is so wrapped up in double-think that it can't even understand itself- it tends to argue too much in favor of "biological differences would mean permanent, uncorrectable injustice, therefore it is impossible that biological differences exist."
- dbt00 7y ago
  > It may well be the case that the gender split is 80% / 20% male to female (as is roughly the case today) Ask the same question in 1965 and get the opposite result. There’s a lot of motivated reasoning by men that their innate characteristics are selected for computer programming. There’s a lot of hostile behavior by men towards women who are in computer science. You can view this like any other resource scarcity turf protection gambit. It’s impossible to separate any apparent belief in the natural order of male dominance of the field from this behavior, so to draw conclusions from it is extremely dangerous.
  - reroute1 7y ago
    > There’s a lot of motivated reasoning by men that their innate characteristics are selected for computer programming. There’s a lot of hostile behavior by men towards women who are in computer science. You can view this like any other resource scarcity turf protection gambit. That is entirely your opinion. Framing biological differences as turf protection is laughable in my opinion.
rumanator 7y ago
> That's the entire point of the article, our biases, as reflected in the world, are learned by models. They are not biased if they represent reality based on facts. In fact, the main issue is that while these algorithms fit reality as it actually is, the critics crying "bias" are complaining that these algorithms don't fit what they believe reality should be. In other words, the algorithms output predictions that matches reality but critics argue that they should instead be manipulated to output results that suits their ideals. But that has nothing to do with having a bias.
- FisDugthop 7y ago
  For example, the model under discussion, which can tell the difference between men and women and believes in differences between them, probably doesn't know that intersex people exist. It probably cannot learn such a distinction, either. Therefore this algorithm probably doesn't fit what we know reality to resemble. So there's at least two sorts of biases. There's both biases inside the model, which is what you're thinking of when you use the word "bias", and also biases outside the model, which inform the model's design and construction in ways that cannot even be quantified with only the model's metrics.
  - rumanator 7y ago
    > For example, the model under discussion, which can tell the difference between men and women and believes in differences between them, probably doesn't know that intersex people exist. Considering that the frequency of intersex people in a population lies somewhere between 0.05% and 0.07%, not accounting for a secondary trait such as whether a biological M or F happens to be intersex is an irrelevant classification error that has a negligible (if any) impact on a model's predictive ability. And by the way, any model can be regenerated if any attribute left out is found to be significant.
- ncallaway 7y ago
  The post you are replying to has three clear bullet points which indicate where the problem lies. It also directly addresses the point you're making about accurate depictions of reality. Would you mind responding to the third bullet point from the parent post, that discusses the dangers of naive and biased applications of facts that are based on reality?
denkmoon 7y ago
Not being educated in anything to do with AI or machine learning, it seems to me that "amplification algorithms" is exactly what these systems are. When I look at a DeepDream image, it appears as though it has amplified many defining characteristics of the original images. Conceptually, reinforcement learning seems very similar to biological positive feedback mechanisms. These mechanisms have guided human development, and cognitive bias is an innate aspect of being human so it stands to reasons that non-biological systems developed in the same way would exhibit biases.
downerending 7y ago
Yes, step three is an abomination of logic. It is not, however, a problem with AI, software, etc. It's entirely the fault of fools at the agency, and perhaps shysters selling them crap software. Garbage in, garbage out.
belorn 7y ago
What is the predictive model predicting, and why would a recruiting agency use those attributes? Any model using nonsense data in predictive model can imminently be ruled out, and I can make a trivial example like. * Most programmers have 10 toes. Fact. * A predictive model correctly identifies that more programmers have 10 toes than 9 toes. * A recruiting agency uses the predictive model and exclude programmers with 9 toes, but why? What possible use could they have with that model? The problem of recruiting agency that using biased predictive models is when the data is relevant for the recruitment purpose but create socially unacceptable conclusions such as: * There are more computer programmers that leave after the first year who identify themselves as women than men. (descriptive statement) * A predictive model use gender as data point to predict employee retention. * An internal recruiting agency within a company has goals of hiring employee with high retention in order to reduce training costs and uses the predictive model. This result in excluding women. The attribute being measured in the later case is relevant for the purpose of the recruiting, but it amplify gender segregation. When discussing AI recruiting bias it is this kind of problem we should focus on rather than looking at attributes which has no bearing on the purpose of recruiting. Attributes which has nothing to do with the goals of recruiting should not exist in a model for recruiting.

by gbrown 7y ago

swebs 7y ago

I'm fail to see how step 2 leads to step 3. The fact that there are less woman programmers overall should not have any affect on P(good|woman,programmer).

crooked-v 7y ago
2 leads to 3 because in a naively designed system, being a man is a stronger predictor of being a good programmer than being a woman, because there are more men than women in high-ranking and high-paying programmer positions.
- deleted 7y ago
  [deleted]
- reroute1 7y ago
  Yes but only in very poorly designed systems. We have those now with AI it's no different. I still see no reason to jump from 2 to 3 other than incompetence in design.
  - loopz 7y ago
    The other poster stated "predictive model". A predictive model could easily identify males to be "more fit" (higher p-value) than females, in aggregate. The programmer tries to evade this bias, by ie. removing gender values in the system. However, the system STILL identify males "more fit", due to correllated values (likes Mancester United/Liverpool). Machine Learning is not much more than advanced statistics, so data biases gets amplified, thus naive application is often a poor fit. Even worse, the experts are at loss on how to make such predictive systems "perform better", without introducing biases on their own, since the data and categorizations themselves may be full of biases. In the real world, incompetence may not be a huge hurdle when selling complex systems. Also, these biases are invisible, until one thinks about them or spots them in the wild.
  - crooked-v 7y ago
    Lots of people think "AI" is a magic bullet that will obviate any need to actually think about this stuff.
- whatshisface 7y ago
  >a stronger predictor of being a good programmer than being a woman Of being a programmer, not being a good programmer. There are more men than women in programming, but the recruiters want to tell good programmers from bad ones, not programmers from nurses. The data clearly shows that a randomly chosen man is more likely to be a programmer than a randomly chosen woman, but that's irrelevant. The likelihood that a randomly chosen female programmer is good should be about the same as the likelihood that a randomly chosen male programmer is good, and that's what hiring managers care about.
  - crooked-v 7y ago
    And yet this kind of problem is happening over and over and over again in real life, such as the automated risk assessments used in Florida deciding that black people are inherently more likely to be criminal than white people. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm...
ska 7y ago
That's a central topic of the article linked, that in practice people are finding it difficult to design and train systems without having this effect.
bluGill 7y ago
If you are randomly selecting people from the general population it follows. That is if you just grab the basic info of unemployed and call some in for a programming interview culling females from the list before scheduling the interview appointment will save you a lot of time before you find/hire a good programmer. I can't figure out why you would use such an algorithm though. Most people who want to hire programmers look at their resumes from people who have self-selected as programmers (possibly on a job search forum) and interview them. If you decide there are not enough programmers you might hire people at random ("would you be willing to learn to be a programmer if paid?" "yes" "hired, you start Monday") - there might be factors to exclude some people because they can never be great programmers, but I'm not aware of what such factors might be - gender doesn't seem to be one though.
ncallaway 7y ago
> I'm fail to see how step 2 leads to step 3. The fact that there are less woman programmers overall should not have any affect on P(good|woman,programmer). You're correct that there should not be any affect. The problem is that many systems are designed such that there is an affect.
- theamk 7y ago
  Are there such systems? Every paper I saw use trivial models which are unlikely to be used in real life. Like this article, which checks gender associations for words like “money” - that’s not how one would be using AI for hiring in any practical way!

Excel_Wizard 7y ago

Let's say that there is a population bell curve of the variable "propensity to enjoy being a programmer". One curve for men, and one curve for women. In some dystopian future, everyone takes a career aptitude test, and if an individual falls within the top 10% of the overall population, they get turned into a programmer by the government. This is assuming a perfectly-trained AI judges the variable, and that it controls against the underlying variables of societal bias in order to ensure that long-term human resources are properly allocated, and that long-term biases, informed by people's interactions with eachother, trend toward actual biological differences. It may well be the case that the gender split is 80% / 20% male to female (as is roughly the case today). It may not. However, the left-leaning zeitgeist opinion would seem to be that this outcome is impossible, and current observed differences are only due to systematic oppression. The right-leaning zeitgeist opinion would seem to be that this outcome would make sense. I tend to think that the left-leaning opinion on this is so wrapped up in double-think that it can't even understand itself- it tends to argue too much in favor of "biological differences would mean permanent, uncorrectable injustice, therefore it is impossible that biological differences exist."

dbt00 7y ago
> It may well be the case that the gender split is 80% / 20% male to female (as is roughly the case today) Ask the same question in 1965 and get the opposite result. There’s a lot of motivated reasoning by men that their innate characteristics are selected for computer programming. There’s a lot of hostile behavior by men towards women who are in computer science. You can view this like any other resource scarcity turf protection gambit. It’s impossible to separate any apparent belief in the natural order of male dominance of the field from this behavior, so to draw conclusions from it is extremely dangerous.
- reroute1 7y ago
  > There’s a lot of motivated reasoning by men that their innate characteristics are selected for computer programming. There’s a lot of hostile behavior by men towards women who are in computer science. You can view this like any other resource scarcity turf protection gambit. That is entirely your opinion. Framing biological differences as turf protection is laughable in my opinion.

rumanator 7y ago

> That's the entire point of the article, our biases, as reflected in the world, are learned by models. They are not biased if they represent reality based on facts. In fact, the main issue is that while these algorithms fit reality as it actually is, the critics crying "bias" are complaining that these algorithms don't fit what they believe reality should be. In other words, the algorithms output predictions that matches reality but critics argue that they should instead be manipulated to output results that suits their ideals. But that has nothing to do with having a bias.

FisDugthop 7y ago
For example, the model under discussion, which can tell the difference between men and women and believes in differences between them, probably doesn't know that intersex people exist. It probably cannot learn such a distinction, either. Therefore this algorithm probably doesn't fit what we know reality to resemble. So there's at least two sorts of biases. There's both biases inside the model, which is what you're thinking of when you use the word "bias", and also biases outside the model, which inform the model's design and construction in ways that cannot even be quantified with only the model's metrics.
- rumanator 7y ago
  > For example, the model under discussion, which can tell the difference between men and women and believes in differences between them, probably doesn't know that intersex people exist. Considering that the frequency of intersex people in a population lies somewhere between 0.05% and 0.07%, not accounting for a secondary trait such as whether a biological M or F happens to be intersex is an irrelevant classification error that has a negligible (if any) impact on a model's predictive ability. And by the way, any model can be regenerated if any attribute left out is found to be significant.
ncallaway 7y ago
The post you are replying to has three clear bullet points which indicate where the problem lies. It also directly addresses the point you're making about accurate depictions of reality. Would you mind responding to the third bullet point from the parent post, that discusses the dangers of naive and biased applications of facts that are based on reality?

denkmoon 7y ago

Not being educated in anything to do with AI or machine learning, it seems to me that "amplification algorithms" is exactly what these systems are. When I look at a DeepDream image, it appears as though it has amplified many defining characteristics of the original images. Conceptually, reinforcement learning seems very similar to biological positive feedback mechanisms. These mechanisms have guided human development, and cognitive bias is an innate aspect of being human so it stands to reasons that non-biological systems developed in the same way would exhibit biases.

downerending 7y ago

Yes, step three is an abomination of logic. It is not, however, a problem with AI, software, etc. It's entirely the fault of fools at the agency, and perhaps shysters selling them crap software. Garbage in, garbage out.

belorn 7y ago

What is the predictive model predicting, and why would a recruiting agency use those attributes? Any model using nonsense data in predictive model can imminently be ruled out, and I can make a trivial example like. * Most programmers have 10 toes. Fact. * A predictive model correctly identifies that more programmers have 10 toes than 9 toes. * A recruiting agency uses the predictive model and exclude programmers with 9 toes, but why? What possible use could they have with that model? The problem of recruiting agency that using biased predictive models is when the data is relevant for the recruitment purpose but create socially unacceptable conclusions such as: * There are more computer programmers that leave after the first year who identify themselves as women than men. (descriptive statement) * A predictive model use gender as data point to predict employee retention. * An internal recruiting agency within a company has goals of hiring employee with high retention in order to reduce training costs and uses the predictive model. This result in excluding women. The attribute being measured in the later case is relevant for the purpose of the recruiting, but it amplify gender segregation. When discussing AI recruiting bias it is this kind of problem we should focus on rather than looking at attributes which has no bearing on the purpose of recruiting. Attributes which has nothing to do with the goals of recruiting should not exist in a model for recruiting.