3 ms·

This is EXACTLY what people are trying to raise awareness of. This is implicit bias. It doesn't matter if people have "good intentions" or made "honest mistakes

This is EXACTLY what people are trying to raise awareness of. This is implicit bias. It doesn't matter if people have "good intentions" or made "honest mistakes" if the tool is implicitly biased. This is why having a diverse team working on a project is important, because it will help call out these issues early rather than after they have been put in production. It's also not purely a technical problem, if you feed the model only pictures of white and asian male college students then it's no surprise when you get a model that biases towards recognizing white and asian male college students (which is exactly how several prominent models were trained).

zerocrates 6y ago
Biased algorithms/models are particularly dangerous because they tend to provide a veneer of objectivity (plausible deniability if you want to put a more cynical lens on it) that could frustrate attempts to hold users accountable.
- dkn775 6y ago
  Agreed. A huge problem I haven't been able to think of is the (already happening in masse) practice of targeting black people because the data says so. However over time police Dept invests data capture resources in "bad areas" - which may have a lot of black residents. I wonder how to separate the defense that some black areas are obviously bad and why would a dept put surveillance resources in a good neighborhood? I personally feel it's wrong but that's one thing I've always got hung up on in building a critique.
cpeterso 6y ago
Mathematician Cathy O'Neil's book "Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy" is a good introduction to the implicit biases in machine learning: https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction
- koheripbal 6y ago
  That book is incredibly misleading. This has nothing to do with machine learning. It is a simple correlational situation. If African Americans have, on average, poorer credit ratings, then correlational models will begin to equate race with poor credit ratings, which will impact their ability to get credit and hence feeding back on that mechanism. ...of course RACE isn't allowed to be factored into financial applications, so the applications will often use other data points, like zip code, that end up having a correlation to bad credit as well as race. ...often producing the same result. The problem isn't with the models - it's with reality. The author famously said "Math is Racist". It's hard to get over such stupidity.
  - mbay 6y ago
    I don't think you're making the case you think you're making.
    - Cogito 6y ago
      It seems like they are making the case that if reality is biased then models of reality will retain that bias. If so they seem to make the point well. If the only information you have about a loan applicant is where they live, your decision will be 'biased' if the location of where someone lives is correlated with other factors (as opposed to, say, the fact they live on a flood plain means don't give them a loan). In this context, saying "Math is Racist" is like saying "Physics hates Fat People" because gravity disproportionately affects heavier people. Accurately reporting what is happening is not biased, making decisions without considering [edit: or not making a decision because you didn't consider] the context is biased. Maths is a tool (well, collection of tools), and the onus is first on the tool user to use it in a fair way. Yes it is important for educators and tool creators to be mindful of how these tools will be used in practice, but there is a big jump from that idea to "Math is Racist".
      - josephh 6y ago
        Isn’t this a similar argument that could be made against race and gender based affirmative action? I don’t understand how organizations like ACLU are critical of face recognition tech because it reenforces implicit bias that engineers have but then turns around and supports race and gender based affirmative action that similarly reenforces implicit bias where PoC (but not Asians for some reasons) and non-males are presumed to be disadvantaged purely due to their identity.

zerocrates 6y ago

Biased algorithms/models are particularly dangerous because they tend to provide a veneer of objectivity (plausible deniability if you want to put a more cynical lens on it) that could frustrate attempts to hold users accountable.

dkn775 6y ago
Agreed. A huge problem I haven't been able to think of is the (already happening in masse) practice of targeting black people because the data says so. However over time police Dept invests data capture resources in "bad areas" - which may have a lot of black residents. I wonder how to separate the defense that some black areas are obviously bad and why would a dept put surveillance resources in a good neighborhood? I personally feel it's wrong but that's one thing I've always got hung up on in building a critique.

cpeterso 6y ago

Mathematician Cathy O'Neil's book "Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy" is a good introduction to the implicit biases in machine learning: https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction

koheripbal 6y ago
That book is incredibly misleading. This has nothing to do with machine learning. It is a simple correlational situation. If African Americans have, on average, poorer credit ratings, then correlational models will begin to equate race with poor credit ratings, which will impact their ability to get credit and hence feeding back on that mechanism. ...of course RACE isn't allowed to be factored into financial applications, so the applications will often use other data points, like zip code, that end up having a correlation to bad credit as well as race. ...often producing the same result. The problem isn't with the models - it's with reality. The author famously said "Math is Racist". It's hard to get over such stupidity.
- mbay 6y ago
  I don't think you're making the case you think you're making.
  - Cogito 6y ago
    It seems like they are making the case that if reality is biased then models of reality will retain that bias. If so they seem to make the point well. If the only information you have about a loan applicant is where they live, your decision will be 'biased' if the location of where someone lives is correlated with other factors (as opposed to, say, the fact they live on a flood plain means don't give them a loan). In this context, saying "Math is Racist" is like saying "Physics hates Fat People" because gravity disproportionately affects heavier people. Accurately reporting what is happening is not biased, making decisions without considering [edit: or not making a decision because you didn't consider] the context is biased. Maths is a tool (well, collection of tools), and the onus is first on the tool user to use it in a fair way. Yes it is important for educators and tool creators to be mindful of how these tools will be used in practice, but there is a big jump from that idea to "Math is Racist".
    - josephh 6y ago
      Isn’t this a similar argument that could be made against race and gender based affirmative action? I don’t understand how organizations like ACLU are critical of face recognition tech because it reenforces implicit bias that engineers have but then turns around and supports race and gender based affirmative action that similarly reenforces implicit bias where PoC (but not Asians for some reasons) and non-males are presumed to be disadvantaged purely due to their identity.