6 ms·

I find that calling it a `racist algorithm` doesn't really do it any good unless the behaviour was intentional. This is a case of poor training data the same as

by nyberg 5y ago

I find that calling it a `racist algorithm` doesn't really do it any good unless the behaviour was intentional. This is a case of poor training data the same as google image classification messing up with tags.

staticshock 5y ago
Plenty of racism in humans isn't malicious, either, but is just a byproduct of bad training data. The outcome is bad regardless of what was intended, and it's the outcome that matters.
- londons_explore 5y ago
  I think plenty of people would say it is the intention that matters far more than the outcome.
  - tobr 5y ago
    Matters how? If someone has the best of intentions but ends up creating a bad outcome, it’s still more important to fix that than to change the opinion of, say, a closeted bigot who has no effect on anyone else. (Yes, real life has more shades of gray than that, and both things are important in practice, because bad intentions don’t tend to lead to good outcomes while good intentions definitely can lead to bad outcomes.) As an aside, “I think plenty of people would say X” is not a very good way to phrase an opinion. It’s ok to say you would say X and argue for it, rather than ascribe the opinion to some undefined group of other people.
sascha_sl 5y ago
Racism as a concept has evolved in meaning. It used to only include the most severe intentional cases of bigoted behavior, whereas now it also includes less obvious biases that lead to preventable but not necessarily intentional instances of everyday prejudice and bigotry. I am for one happy we have unneutered the word from having to reach a bar so high, it wouldn't apply to most bigotry, but it is also unfortunate for people who have not caught on and believe calling a thing racist is a damning statement of evil intent, but it really is not anymore. Or those that insist on meaning of words remaining static forever.
heavyset_go 5y ago
Let's say your company decides to use AI to assist in hiring. It turns out the algorithm used is biased when it comes to candidates' race. If there is a disparate impact[1] on protected classes in hiring that's unrelated to job performance, intentions don't matter in the eyes of labor law, what matters are the effects. [1] https://en.wikipedia.org/wiki/Disparate_impact https://en.wikipedia.org/wiki/Disparate_impact
bryanrasmussen 5y ago
>This is a case of poor training data ok we need pictures of human faces, luckily I've got all these white people here! on edit: it was racist in result, in that it empowers a racist system, it was not racist in intention - as in the people gathering the training data probably didn't say hey how can we empower a racist system with this?
- codeulike 5y ago
  yes exactly
- MikeUt 5y ago
  So were those white people exploited to develop Twitter's algorithm? Did the use of their faces raise concerns about the ethics of data harvesting? Google exploited homeless black people to develop the Pixel 4’s facial recognition AI - https://thenextweb.com/news/google-exploited-homeless-black-people-to-develop-the-pixel-4s-facial-recognition-ai https://thenextweb.com/news/google-exploited-homeless-black-... an attempt by Google to improve its facial recognition algorithms by collecting data from people with dark skin is raising further concerns about the ethics of the data harvesting - https://www.theguardian.com/technology/2019/oct/03/google-data-harvesting-facial-recognition-people-of-color https://www.theguardian.com/technology/2019/oct/03/google-da...

by nyberg 5y ago

staticshock 5y ago

Plenty of racism in humans isn't malicious, either, but is just a byproduct of bad training data. The outcome is bad regardless of what was intended, and it's the outcome that matters.

londons_explore 5y ago
I think plenty of people would say it is the intention that matters far more than the outcome.
- tobr 5y ago
  Matters how? If someone has the best of intentions but ends up creating a bad outcome, it’s still more important to fix that than to change the opinion of, say, a closeted bigot who has no effect on anyone else. (Yes, real life has more shades of gray than that, and both things are important in practice, because bad intentions don’t tend to lead to good outcomes while good intentions definitely can lead to bad outcomes.) As an aside, “I think plenty of people would say X” is not a very good way to phrase an opinion. It’s ok to say you would say X and argue for it, rather than ascribe the opinion to some undefined group of other people.

sascha_sl 5y ago

Racism as a concept has evolved in meaning. It used to only include the most severe intentional cases of bigoted behavior, whereas now it also includes less obvious biases that lead to preventable but not necessarily intentional instances of everyday prejudice and bigotry. I am for one happy we have unneutered the word from having to reach a bar so high, it wouldn't apply to most bigotry, but it is also unfortunate for people who have not caught on and believe calling a thing racist is a damning statement of evil intent, but it really is not anymore. Or those that insist on meaning of words remaining static forever.

heavyset_go 5y ago

Let's say your company decides to use AI to assist in hiring. It turns out the algorithm used is biased when it comes to candidates' race. If there is a disparate impact[1] on protected classes in hiring that's unrelated to job performance, intentions don't matter in the eyes of labor law, what matters are the effects. [1] https://en.wikipedia.org/wiki/Disparate_impact https://en.wikipedia.org/wiki/Disparate_impact

bryanrasmussen 5y ago

>This is a case of poor training data ok we need pictures of human faces, luckily I've got all these white people here! on edit: it was racist in result, in that it empowers a racist system, it was not racist in intention - as in the people gathering the training data probably didn't say hey how can we empower a racist system with this?

codeulike 5y ago
yes exactly
MikeUt 5y ago
So were those white people exploited to develop Twitter's algorithm? Did the use of their faces raise concerns about the ethics of data harvesting? Google exploited homeless black people to develop the Pixel 4’s facial recognition AI - https://thenextweb.com/news/google-exploited-homeless-black-people-to-develop-the-pixel-4s-facial-recognition-ai https://thenextweb.com/news/google-exploited-homeless-black-... an attempt by Google to improve its facial recognition algorithms by collecting data from people with dark skin is raising further concerns about the ethics of the data harvesting - https://www.theguardian.com/technology/2019/oct/03/google-data-harvesting-facial-recognition-people-of-color https://www.theguardian.com/technology/2019/oct/03/google-da...