5 ms·

Algorithms are not free of -isms, they just represent the -isms of the person who came up with that algorithm.

by TonyTrapp 5y ago

TheFreim 5y ago
Are there people programming their "algorithms" with flags that say "if black, fire them"???
- TonyTrapp 5y ago
  That's not even necessary. Some of those -isms creep in implicitly, because e.g. certain performance metrics assumed by the algorithm may be statistically less likely to be fulfilled by women, or stuff like that.
  - NalNezumi 5y ago
    Doesn't this make it an intractable problem though? Attribute X is more common in group Y first require a population based study, which can be tampered left and right and doesn't always stay the same over time. (commonality of the attribute can shift with generations) Besides if the goal is "make an algorithm that only base its judgment on population studies" all human decision making algorithms will be the same, and you can endlessly cry wolf.
- lostcolony 5y ago
  Probably not, but so what? It happens all the time, because the non-racist logic still affects people unequally due to racial reasons. Here's the top result from "racist algorithms" in Google - https://www.nature.com/articles/d41586-019-03228-6 https://www.nature.com/articles/d41586-019-03228-6 TL;DR: One of the factors for referring people to treatment was doctor visits over the last year. Makes sense; unwell people go to the doctor, so those going to the doctor more often are more likely to be unwell. But, of course, due to historical segregation, generational wealth, etc, black people on average tended to go to the doctor less. So even if they exhibited the same risks, symptoms, etc, as white people, because they saw a doctor about it less often the algorithm was less likely to refer them for treatment.
- scottLobster 5y ago
  No, by they might filter for traits like less likely to commit a crime. Given that African Americans commit roughly half of all crime in America (last time I read the FBI data), such an algorithm would get really racist, really fast if left to itself.
  - pseudalopex 5y ago
    Black people made up 26.6% of 2019 arrests according to FBI data.[1] And arrests and crimes aren't the same. Few wage thieves are arrested for example. [1] https://ucr.fbi.gov/crime-in-the-u.s/2019/crime-in-the-u.s.-2019/topic-pages/tables/table-43 https://ucr.fbi.gov/crime-in-the-u.s/2019/crime-in-the-u.s.-...
- AlexandrB 5y ago
  I mean, kind of. If you develop an algorithm and the user you're picturing in your mind looks like you, talks like you, and lives like you do you're embedding all kind of gender-related, racial, and socioeconomic biases. The most famous example being automatic soap dispensers that don't detect black hands[1]. Now combine that fact with the demographics of Silicon Valley companies and you get a recipe for products and algorithms that are "fair" only if you match a set of implicit criteria. None of this is intentionally malevolent, just a product of hubris and "move fast and break things". [1] https://reporter.rit.edu/tech/bigotry-encoded-racial-bias-technology https://reporter.rit.edu/tech/bigotry-encoded-racial-bias-te...
  - TheFreim 5y ago
    Doesn't having someone who doesn't speak like you, doesn't share your culture, etc make working together worse and therefore not as good of a match? If I went to China and tried to get a job I wouldn't fault them for choosing someone or equal qualification who's also Chinese since by that very fact they'd be a better team fit. Does that make sense?
theamk 5y ago
A lot of -isms are "hidden" -- for example, a manager might be saying to everyone that they do not discriminate, but in practice, they'll never choose people of certain race for promotions. For human managers, this can be hard to prove, especially if they do not promote people very often. But for algorithms, it is much easier -- multiple people can inspect the source code for suspicious logic. And even if this is some sort of impenetrable AI model, you can still run it with different inputs, and check if the output changes if you change the input race.
Mordisquitos 5y ago
I propose we extend the definition of algorism: al•go•rism /ˈælɡəɹɪzəm/ n. 1. Obsolete spelling of algorithm. 2. An irrational belief in the infallibility of algorithms.
MontyCarloHall 5y ago
They almost certainly will not encode any explicit biases of the algorithm authors (e.g. “she’s pregnant, can we get rid of her?”), which is already a great step forward, as explicit bias is still sadly far too common in the workplace. When implicit biases find their way into algorithms, it is usually a specific edge case that’s a function of the training data (e.g. facial recognition preforming poorly on darker skinned people due to training set imbalance). The biggest case of implicit bias in algorithms I’m aware of is when the training set is the behavior of society as a whole, as with search engines: for example, women search less for C-level jobs and so are less likely to be shown postings for C-level jobs when doing job searches (thereby perpetuating a vicious cycle), or websites depicting Black teens are more likely to show them in a criminal context than White teens, so searches for “black teenager” show pictures of criminals whereas searches for “white teenager” do not. These implicit biases are not encoded by the algorithm developer but rather by the dataset the algorithm is applied to. In the case of automated firing, I don’t see how implicit bias can creep in if the metrics are strictly work-related (e.g. fraction of on-time deliveries). For the record, I do not agree with automating performance metrics, since they cannot account for nuance (e.g. delivery drivers assigned to gated communities have issues opening the gate and thus deliver fewer on-time packages, which is not accounted for in the algorithm.) However, this is not a form of demographic bias, explicit or implicit.
- jeromegv 5y ago
  Where I think you are wrong here is that you fail to consider the fact that the same programmer must decide which dataset/training data they will use for the AI. So implicit biais is not just external to the programmer, it's definitely caused by the programmer. And implicit biais goes way beyond their training dataset. Did they put a clause "if pregnant, then...", likely not. However, what conditions did they use for testing this algorithm and ensure if it works? Testing is really much part of the process as well.
  - MontyCarloHall 5y ago
    > So implicit biais is not just external to the programmer, it's definitely caused by the programmer. I agree in the case of things like facial recognition, but in the case of search results, Google can’t exactly choose whether websites it indexes present “Black teens” in a poor light and “White teens” in a good light. Perfectly unbiased algorithms trained on data produced by society will always exactly reflect the unfortunate biases of society. (Ironically, attempting to correct for society’s biases in the algorithm would by definition bias the algorithm itself!) This is (likely) not the case with Amazon drivers, since the training data are (presumably) strictly job performance-related, e.g. fraction of on-time deliveries, or number of times a worker showed up late. And even if these happen to correlate to demographics (e.g. if Black workers lived further from the warehouse than White workers [or vice-versa] and were thus more likely to show up to work late), tracking employees based on their on-time performance is not bias on Amazon’s part, since a White worker who showed up late would be just as likely to be fired as a Black worker who showed up late. The fact that one demographic (by no fault of its own) happens to show up late more often is not bias on the part of Amazon.

Algorithms are not free of -isms, they just represent the -isms of the person who came up with that algorithm.

TheFreim 5y ago

Are there people programming their "algorithms" with flags that say "if black, fire them"???

TonyTrapp 5y ago
That's not even necessary. Some of those -isms creep in implicitly, because e.g. certain performance metrics assumed by the algorithm may be statistically less likely to be fulfilled by women, or stuff like that.
- NalNezumi 5y ago
  Doesn't this make it an intractable problem though? Attribute X is more common in group Y first require a population based study, which can be tampered left and right and doesn't always stay the same over time. (commonality of the attribute can shift with generations) Besides if the goal is "make an algorithm that only base its judgment on population studies" all human decision making algorithms will be the same, and you can endlessly cry wolf.
lostcolony 5y ago
Probably not, but so what? It happens all the time, because the non-racist logic still affects people unequally due to racial reasons. Here's the top result from "racist algorithms" in Google - https://www.nature.com/articles/d41586-019-03228-6 https://www.nature.com/articles/d41586-019-03228-6 TL;DR: One of the factors for referring people to treatment was doctor visits over the last year. Makes sense; unwell people go to the doctor, so those going to the doctor more often are more likely to be unwell. But, of course, due to historical segregation, generational wealth, etc, black people on average tended to go to the doctor less. So even if they exhibited the same risks, symptoms, etc, as white people, because they saw a doctor about it less often the algorithm was less likely to refer them for treatment.
scottLobster 5y ago
No, by they might filter for traits like less likely to commit a crime. Given that African Americans commit roughly half of all crime in America (last time I read the FBI data), such an algorithm would get really racist, really fast if left to itself.
- pseudalopex 5y ago
  Black people made up 26.6% of 2019 arrests according to FBI data.[1] And arrests and crimes aren't the same. Few wage thieves are arrested for example. [1] https://ucr.fbi.gov/crime-in-the-u.s/2019/crime-in-the-u.s.-2019/topic-pages/tables/table-43 https://ucr.fbi.gov/crime-in-the-u.s/2019/crime-in-the-u.s.-...
AlexandrB 5y ago
I mean, kind of. If you develop an algorithm and the user you're picturing in your mind looks like you, talks like you, and lives like you do you're embedding all kind of gender-related, racial, and socioeconomic biases. The most famous example being automatic soap dispensers that don't detect black hands[1]. Now combine that fact with the demographics of Silicon Valley companies and you get a recipe for products and algorithms that are "fair" only if you match a set of implicit criteria. None of this is intentionally malevolent, just a product of hubris and "move fast and break things". [1] https://reporter.rit.edu/tech/bigotry-encoded-racial-bias-technology https://reporter.rit.edu/tech/bigotry-encoded-racial-bias-te...
- TheFreim 5y ago
  Doesn't having someone who doesn't speak like you, doesn't share your culture, etc make working together worse and therefore not as good of a match? If I went to China and tried to get a job I wouldn't fault them for choosing someone or equal qualification who's also Chinese since by that very fact they'd be a better team fit. Does that make sense?

theamk 5y ago

A lot of -isms are "hidden" -- for example, a manager might be saying to everyone that they do not discriminate, but in practice, they'll never choose people of certain race for promotions. For human managers, this can be hard to prove, especially if they do not promote people very often. But for algorithms, it is much easier -- multiple people can inspect the source code for suspicious logic. And even if this is some sort of impenetrable AI model, you can still run it with different inputs, and check if the output changes if you change the input race.

Mordisquitos 5y ago

I propose we extend the definition of algorism: al•go•rism /ˈælɡəɹɪzəm/ n. 1. Obsolete spelling of algorithm. 2. An irrational belief in the infallibility of algorithms.

MontyCarloHall 5y ago

They almost certainly will not encode any explicit biases of the algorithm authors (e.g. “she’s pregnant, can we get rid of her?”), which is already a great step forward, as explicit bias is still sadly far too common in the workplace. When implicit biases find their way into algorithms, it is usually a specific edge case that’s a function of the training data (e.g. facial recognition preforming poorly on darker skinned people due to training set imbalance). The biggest case of implicit bias in algorithms I’m aware of is when the training set is the behavior of society as a whole, as with search engines: for example, women search less for C-level jobs and so are less likely to be shown postings for C-level jobs when doing job searches (thereby perpetuating a vicious cycle), or websites depicting Black teens are more likely to show them in a criminal context than White teens, so searches for “black teenager” show pictures of criminals whereas searches for “white teenager” do not. These implicit biases are not encoded by the algorithm developer but rather by the dataset the algorithm is applied to. In the case of automated firing, I don’t see how implicit bias can creep in if the metrics are strictly work-related (e.g. fraction of on-time deliveries). For the record, I do not agree with automating performance metrics, since they cannot account for nuance (e.g. delivery drivers assigned to gated communities have issues opening the gate and thus deliver fewer on-time packages, which is not accounted for in the algorithm.) However, this is not a form of demographic bias, explicit or implicit.

jeromegv 5y ago
Where I think you are wrong here is that you fail to consider the fact that the same programmer must decide which dataset/training data they will use for the AI. So implicit biais is not just external to the programmer, it's definitely caused by the programmer. And implicit biais goes way beyond their training dataset. Did they put a clause "if pregnant, then...", likely not. However, what conditions did they use for testing this algorithm and ensure if it works? Testing is really much part of the process as well.
- MontyCarloHall 5y ago
  > So implicit biais is not just external to the programmer, it's definitely caused by the programmer. I agree in the case of things like facial recognition, but in the case of search results, Google can’t exactly choose whether websites it indexes present “Black teens” in a poor light and “White teens” in a good light. Perfectly unbiased algorithms trained on data produced by society will always exactly reflect the unfortunate biases of society. (Ironically, attempting to correct for society’s biases in the algorithm would by definition bias the algorithm itself!) This is (likely) not the case with Amazon drivers, since the training data are (presumably) strictly job performance-related, e.g. fraction of on-time deliveries, or number of times a worker showed up late. And even if these happen to correlate to demographics (e.g. if Black workers lived further from the warehouse than White workers [or vice-versa] and were thus more likely to show up to work late), tracking employees based on their on-time performance is not bias on Amazon’s part, since a White worker who showed up late would be just as likely to be fired as a Black worker who showed up late. The fact that one demographic (by no fault of its own) happens to show up late more often is not bias on the part of Amazon.