4 ms·

Maybe I’m naive or just dumb, but: isn’t this better? One of the big complaints about modern society is all the isms. Sexism, racism, homophobia (ism). Compu

by timmg 5y ago

Maybe I’m naive or just dumb, but: isn’t this better? One of the big complaints about modern society is all the isms. Sexism, racism, homophobia (ism). Computers don’t know your skin color, gender, religion, sexual preference, etc. Not saying it will solve all problems, but wouldn’t algorithmic decisions be preferable to biased human judgement?

TonyTrapp 5y ago
Algorithms are not free of -isms, they just represent the -isms of the person who came up with that algorithm.
- TheFreim 5y ago
  Are there people programming their "algorithms" with flags that say "if black, fire them"???
  - TonyTrapp 5y ago
    That's not even necessary. Some of those -isms creep in implicitly, because e.g. certain performance metrics assumed by the algorithm may be statistically less likely to be fulfilled by women, or stuff like that.
    - NalNezumi 5y ago
      Doesn't this make it an intractable problem though? Attribute X is more common in group Y first require a population based study, which can be tampered left and right and doesn't always stay the same over time. (commonality of the attribute can shift with generations) Besides if the goal is "make an algorithm that only base its judgment on population studies" all human decision making algorithms will be the same, and you can endlessly cry wolf.
  - lostcolony 5y ago
    Probably not, but so what? It happens all the time, because the non-racist logic still affects people unequally due to racial reasons. Here's the top result from "racist algorithms" in Google - https://www.nature.com/articles/d41586-019-03228-6 https://www.nature.com/articles/d41586-019-03228-6 TL;DR: One of the factors for referring people to treatment was doctor visits over the last year. Makes sense; unwell people go to the doctor, so those going to the doctor more often are more likely to be unwell. But, of course, due to historical segregation, generational wealth, etc, black people on average tended to go to the doctor less. So even if they exhibited the same risks, symptoms, etc, as white people, because they saw a doctor about it less often the algorithm was less likely to refer them for treatment.
  - scottLobster 5y ago
    No, by they might filter for traits like less likely to commit a crime. Given that African Americans commit roughly half of all crime in America (last time I read the FBI data), such an algorithm would get really racist, really fast if left to itself.
    - pseudalopex 5y ago
      Black people made up 26.6% of 2019 arrests according to FBI data.[1] And arrests and crimes aren't the same. Few wage thieves are arrested for example. [1] https://ucr.fbi.gov/crime-in-the-u.s/2019/crime-in-the-u.s.-2019/topic-pages/tables/table-43 https://ucr.fbi.gov/crime-in-the-u.s/2019/crime-in-the-u.s.-...
  - AlexandrB 5y ago
    I mean, kind of. If you develop an algorithm and the user you're picturing in your mind looks like you, talks like you, and lives like you do you're embedding all kind of gender-related, racial, and socioeconomic biases. The most famous example being automatic soap dispensers that don't detect black hands[1]. Now combine that fact with the demographics of Silicon Valley companies and you get a recipe for products and algorithms that are "fair" only if you match a set of implicit criteria. None of this is intentionally malevolent, just a product of hubris and "move fast and break things". [1] https://reporter.rit.edu/tech/bigotry-encoded-racial-bias-technology https://reporter.rit.edu/tech/bigotry-encoded-racial-bias-te...
    - TheFreim 5y ago
      Doesn't having someone who doesn't speak like you, doesn't share your culture, etc make working together worse and therefore not as good of a match? If I went to China and tried to get a job I wouldn't fault them for choosing someone or equal qualification who's also Chinese since by that very fact they'd be a better team fit. Does that make sense?
- theamk 5y ago
  A lot of -isms are "hidden" -- for example, a manager might be saying to everyone that they do not discriminate, but in practice, they'll never choose people of certain race for promotions. For human managers, this can be hard to prove, especially if they do not promote people very often. But for algorithms, it is much easier -- multiple people can inspect the source code for suspicious logic. And even if this is some sort of impenetrable AI model, you can still run it with different inputs, and check if the output changes if you change the input race.
- Mordisquitos 5y ago
  I propose we extend the definition of algorism: al•go•rism /ˈælɡəɹɪzəm/ n. 1. Obsolete spelling of algorithm. 2. An irrational belief in the infallibility of algorithms.
- MontyCarloHall 5y ago
  They almost certainly will not encode any explicit biases of the algorithm authors (e.g. “she’s pregnant, can we get rid of her?”), which is already a great step forward, as explicit bias is still sadly far too common in the workplace. When implicit biases find their way into algorithms, it is usually a specific edge case that’s a function of the training data (e.g. facial recognition preforming poorly on darker skinned people due to training set imbalance). The biggest case of implicit bias in algorithms I’m aware of is when the training set is the behavior of society as a whole, as with search engines: for example, women search less for C-level jobs and so are less likely to be shown postings for C-level jobs when doing job searches (thereby perpetuating a vicious cycle), or websites depicting Black teens are more likely to show them in a criminal context than White teens, so searches for “black teenager” show pictures of criminals whereas searches for “white teenager” do not. These implicit biases are not encoded by the algorithm developer but rather by the dataset the algorithm is applied to. In the case of automated firing, I don’t see how implicit bias can creep in if the metrics are strictly work-related (e.g. fraction of on-time deliveries). For the record, I do not agree with automating performance metrics, since they cannot account for nuance (e.g. delivery drivers assigned to gated communities have issues opening the gate and thus deliver fewer on-time packages, which is not accounted for in the algorithm.) However, this is not a form of demographic bias, explicit or implicit.
  - jeromegv 5y ago
    Where I think you are wrong here is that you fail to consider the fact that the same programmer must decide which dataset/training data they will use for the AI. So implicit biais is not just external to the programmer, it's definitely caused by the programmer. And implicit biais goes way beyond their training dataset. Did they put a clause "if pregnant, then...", likely not. However, what conditions did they use for testing this algorithm and ensure if it works? Testing is really much part of the process as well.
    - MontyCarloHall 5y ago
      > So implicit biais is not just external to the programmer, it's definitely caused by the programmer. I agree in the case of things like facial recognition, but in the case of search results, Google can’t exactly choose whether websites it indexes present “Black teens” in a poor light and “White teens” in a good light. Perfectly unbiased algorithms trained on data produced by society will always exactly reflect the unfortunate biases of society. (Ironically, attempting to correct for society’s biases in the algorithm would by definition bias the algorithm itself!) This is (likely) not the case with Amazon drivers, since the training data are (presumably) strictly job performance-related, e.g. fraction of on-time deliveries, or number of times a worker showed up late. And even if these happen to correlate to demographics (e.g. if Black workers lived further from the warehouse than White workers [or vice-versa] and were thus more likely to show up to work late), tracking employees based on their on-time performance is not bias on Amazon’s part, since a White worker who showed up late would be just as likely to be fired as a Black worker who showed up late. The fact that one demographic (by no fault of its own) happens to show up late more often is not bias on the part of Amazon.
jameskilton 5y ago
I would suggest reading up on why people are getting fired. e.g. getting fired because you couldn't fulfill your "quota" because some neighborhoods have their gates locked over the weekend. Imagine getting fired for something you have no control over, or didn't even realize was getting penalized for. This system is cruelty, pure and simple. Is it any wonder that Amazon execs are worried about running out of people to hire?
- matz1 5y ago
  >Imagine getting fired for something you have no control over, or didn't even realize was getting penalized for. That is life, that could always happen, the right thing is to accept the fact and plan accordingly. If i took any kind of job I acknowledge that there always possibility that I'm going to be fired for something I have no control.
  - scottLobster 5y ago
    Sure, but I can't imagine you'd plan for being fired for being locked out of your work area. There's a difference between having your house blow down by a hurricane vs having your house blown down by a slight breeze, and this would appear to be the latter. Also, even in the US, you can sue for wrongful termination depending on circumstances.
    - matz1 5y ago
      >Sure, but I can't imagine you'd plan for being fired for being locked out of your work area The reason of being fired is beside the point. Employer should be able to fire anyone for any reason. Likewise employee should be able to quit for any reason.
      - prower 5y ago
        "You're free to quit" doesn't sound like the most correct groundwork to build an honest society.
- 5y ago
mschuster91 5y ago
> but wouldn’t algorithmic decisions be preferable to biased human judgement? An algorithm also can't apply correctional factors (e.g. age, physical strength, disabilities).
- prower 5y ago
  Not to mention the elephant in the room that an over-optimized environment is not suitable for human life.
- trey-jones 5y ago
  I think the point of the OP is that these factors don't matter. Performance is the only goal. In general I think this is fine, but it's clear that the algorithm is not yet sound. Maybe we'll get there one day.
  - kayodelycaon 5y ago
    In America, ADA absolutely matters. Algorithms like this can blatantly violate the law and open the company to significant liabilities. I've seen a company repeatedly lose expense lawsuits due to "following process" where the process did not take federal laws protecting workers into account.
WesolyKubeczek 5y ago
Such algorithms usually emphasize biases instead of dissipating them. Sometimes I think though, maybe we should admit in the longer run that some of these biases are actually the truth and all talk of "fair" and "unbiased" is some kind of wishful thinking against all the facts that belongs to the land of teletubbies.
trabant00 5y ago
> wouldn’t algorithmic decisions be preferable to biased human judgement? At this moment? Big NO. Algorithms - are programmed by humans - trained on past data (which by definition includes all the "isms") - have a gajillion corner cases in which they perform hilariously bad - don't understand the data they are processing so they can't reason something must be garbage in How's that Amazon recommendation system working out for you compared to a human recommendation? That's the current state of the art algorithm on a very simple problem.
- thebean11 5y ago
  I kind of doubt Amazon's algorithm even uses machine learning, it probably assigns a score based on perf metrics (using a human-written algorithm), then fires employees below some cutoff.
- thefounder 5y ago
  Do you really need an AI to measure the performance(i.e require training as you said) ? I thibk a dumb counter would do the job.
  - playpause 5y ago
    Who mentioned AI?
    - thefounder 5y ago
      What is "- trained on past data (which by definition includes all the "isms")" supposed to mean?
- mc32 5y ago
  >” trained on past data (which by definition includes all the "isms")” Is that necessarily so? Can’t they measure more objective data points like deliveries per hour, missdeliveries, mishandled/broken items (controlled for traffic, pop density, data from deliveries to similar areas, etc) rather than other things like “doesn’t get along with boss:peers, has bad breath, does cat-calls, etc? Now, I admit this has too much Taylorism in it. To much of squeezing the last drop out of people, but I don’t think it has more soft bias.
enumjorge 5y ago
Aside from what others have already replied, I think about how bad automated moderation on big platforms can be, and it doesn’t give me high hopes for this type of software. Think of the stories of an overly aggressive algorithm banning someone for unclear reasons. This has happened to people’s Google accounts. Since Google has limited customer support, people often have a hard time contending the decision. They are locked out of an important slice of their digital lives (email, photos, etc). Can you imagine this happening except it’s your employer who fired you?
maxehmookau 5y ago
You're neither naive or dumb, but this is peak HN. No, it's not better. Algorithms aren't some magical unbiased utopia. They're created by people, with their own biases. They just add a level of separation between the human bias and the person getting fired. Humans are, at least, accountable to other humans.
2pEXgD0fZ5cF 5y ago
> biased human judgement This is the problem, the software is just trained by human decisions. So in many cases they are set up to become even worse. Generally it is bias cast into software. The worst part is that they manage to give the false impression of neutrality. And that doesn't even include all the other problems (cases being ignored/overlooked, false analysis and errors etc.). Also who guarantees that neutral/fair treatment is ever the actual goal?
wyager 5y ago
Your mistake is in assuming uniform treatment is actually the goal. People have mostly publicly given up that pretense, and now there are plenty of explanations for why fairness isn’t enough and we need to actually have a preference for <group> to <do moral thing>. And even that is just a pretense at some level; most of this stuff is actually politics that has little to do with its stated or even first-order-revealed preference.

by timmg 5y ago

TonyTrapp 5y ago

Algorithms are not free of -isms, they just represent the -isms of the person who came up with that algorithm.

TheFreim 5y ago
Are there people programming their "algorithms" with flags that say "if black, fire them"???
- TonyTrapp 5y ago
  That's not even necessary. Some of those -isms creep in implicitly, because e.g. certain performance metrics assumed by the algorithm may be statistically less likely to be fulfilled by women, or stuff like that.
  - NalNezumi 5y ago
    Doesn't this make it an intractable problem though? Attribute X is more common in group Y first require a population based study, which can be tampered left and right and doesn't always stay the same over time. (commonality of the attribute can shift with generations) Besides if the goal is "make an algorithm that only base its judgment on population studies" all human decision making algorithms will be the same, and you can endlessly cry wolf.
- lostcolony 5y ago
  Probably not, but so what? It happens all the time, because the non-racist logic still affects people unequally due to racial reasons. Here's the top result from "racist algorithms" in Google - https://www.nature.com/articles/d41586-019-03228-6 https://www.nature.com/articles/d41586-019-03228-6 TL;DR: One of the factors for referring people to treatment was doctor visits over the last year. Makes sense; unwell people go to the doctor, so those going to the doctor more often are more likely to be unwell. But, of course, due to historical segregation, generational wealth, etc, black people on average tended to go to the doctor less. So even if they exhibited the same risks, symptoms, etc, as white people, because they saw a doctor about it less often the algorithm was less likely to refer them for treatment.
- scottLobster 5y ago
  No, by they might filter for traits like less likely to commit a crime. Given that African Americans commit roughly half of all crime in America (last time I read the FBI data), such an algorithm would get really racist, really fast if left to itself.
  - pseudalopex 5y ago
    Black people made up 26.6% of 2019 arrests according to FBI data.[1] And arrests and crimes aren't the same. Few wage thieves are arrested for example. [1] https://ucr.fbi.gov/crime-in-the-u.s/2019/crime-in-the-u.s.-2019/topic-pages/tables/table-43 https://ucr.fbi.gov/crime-in-the-u.s/2019/crime-in-the-u.s.-...
- AlexandrB 5y ago
  I mean, kind of. If you develop an algorithm and the user you're picturing in your mind looks like you, talks like you, and lives like you do you're embedding all kind of gender-related, racial, and socioeconomic biases. The most famous example being automatic soap dispensers that don't detect black hands[1]. Now combine that fact with the demographics of Silicon Valley companies and you get a recipe for products and algorithms that are "fair" only if you match a set of implicit criteria. None of this is intentionally malevolent, just a product of hubris and "move fast and break things". [1] https://reporter.rit.edu/tech/bigotry-encoded-racial-bias-technology https://reporter.rit.edu/tech/bigotry-encoded-racial-bias-te...
  - TheFreim 5y ago
    Doesn't having someone who doesn't speak like you, doesn't share your culture, etc make working together worse and therefore not as good of a match? If I went to China and tried to get a job I wouldn't fault them for choosing someone or equal qualification who's also Chinese since by that very fact they'd be a better team fit. Does that make sense?
theamk 5y ago
A lot of -isms are "hidden" -- for example, a manager might be saying to everyone that they do not discriminate, but in practice, they'll never choose people of certain race for promotions. For human managers, this can be hard to prove, especially if they do not promote people very often. But for algorithms, it is much easier -- multiple people can inspect the source code for suspicious logic. And even if this is some sort of impenetrable AI model, you can still run it with different inputs, and check if the output changes if you change the input race.
Mordisquitos 5y ago
I propose we extend the definition of algorism: al•go•rism /ˈælɡəɹɪzəm/ n. 1. Obsolete spelling of algorithm. 2. An irrational belief in the infallibility of algorithms.
MontyCarloHall 5y ago
They almost certainly will not encode any explicit biases of the algorithm authors (e.g. “she’s pregnant, can we get rid of her?”), which is already a great step forward, as explicit bias is still sadly far too common in the workplace. When implicit biases find their way into algorithms, it is usually a specific edge case that’s a function of the training data (e.g. facial recognition preforming poorly on darker skinned people due to training set imbalance). The biggest case of implicit bias in algorithms I’m aware of is when the training set is the behavior of society as a whole, as with search engines: for example, women search less for C-level jobs and so are less likely to be shown postings for C-level jobs when doing job searches (thereby perpetuating a vicious cycle), or websites depicting Black teens are more likely to show them in a criminal context than White teens, so searches for “black teenager” show pictures of criminals whereas searches for “white teenager” do not. These implicit biases are not encoded by the algorithm developer but rather by the dataset the algorithm is applied to. In the case of automated firing, I don’t see how implicit bias can creep in if the metrics are strictly work-related (e.g. fraction of on-time deliveries). For the record, I do not agree with automating performance metrics, since they cannot account for nuance (e.g. delivery drivers assigned to gated communities have issues opening the gate and thus deliver fewer on-time packages, which is not accounted for in the algorithm.) However, this is not a form of demographic bias, explicit or implicit.
- jeromegv 5y ago
  Where I think you are wrong here is that you fail to consider the fact that the same programmer must decide which dataset/training data they will use for the AI. So implicit biais is not just external to the programmer, it's definitely caused by the programmer. And implicit biais goes way beyond their training dataset. Did they put a clause "if pregnant, then...", likely not. However, what conditions did they use for testing this algorithm and ensure if it works? Testing is really much part of the process as well.
  - MontyCarloHall 5y ago
    > So implicit biais is not just external to the programmer, it's definitely caused by the programmer. I agree in the case of things like facial recognition, but in the case of search results, Google can’t exactly choose whether websites it indexes present “Black teens” in a poor light and “White teens” in a good light. Perfectly unbiased algorithms trained on data produced by society will always exactly reflect the unfortunate biases of society. (Ironically, attempting to correct for society’s biases in the algorithm would by definition bias the algorithm itself!) This is (likely) not the case with Amazon drivers, since the training data are (presumably) strictly job performance-related, e.g. fraction of on-time deliveries, or number of times a worker showed up late. And even if these happen to correlate to demographics (e.g. if Black workers lived further from the warehouse than White workers [or vice-versa] and were thus more likely to show up to work late), tracking employees based on their on-time performance is not bias on Amazon’s part, since a White worker who showed up late would be just as likely to be fired as a Black worker who showed up late. The fact that one demographic (by no fault of its own) happens to show up late more often is not bias on the part of Amazon.

jameskilton 5y ago

I would suggest reading up on why people are getting fired. e.g. getting fired because you couldn't fulfill your "quota" because some neighborhoods have their gates locked over the weekend. Imagine getting fired for something you have no control over, or didn't even realize was getting penalized for. This system is cruelty, pure and simple. Is it any wonder that Amazon execs are worried about running out of people to hire?

matz1 5y ago
>Imagine getting fired for something you have no control over, or didn't even realize was getting penalized for. That is life, that could always happen, the right thing is to accept the fact and plan accordingly. If i took any kind of job I acknowledge that there always possibility that I'm going to be fired for something I have no control.
- scottLobster 5y ago
  Sure, but I can't imagine you'd plan for being fired for being locked out of your work area. There's a difference between having your house blow down by a hurricane vs having your house blown down by a slight breeze, and this would appear to be the latter. Also, even in the US, you can sue for wrongful termination depending on circumstances.
  - matz1 5y ago
    >Sure, but I can't imagine you'd plan for being fired for being locked out of your work area The reason of being fired is beside the point. Employer should be able to fire anyone for any reason. Likewise employee should be able to quit for any reason.
    - prower 5y ago
      "You're free to quit" doesn't sound like the most correct groundwork to build an honest society.
5y ago

mschuster91 5y ago

> but wouldn’t algorithmic decisions be preferable to biased human judgement? An algorithm also can't apply correctional factors (e.g. age, physical strength, disabilities).

prower 5y ago
Not to mention the elephant in the room that an over-optimized environment is not suitable for human life.
trey-jones 5y ago
I think the point of the OP is that these factors don't matter. Performance is the only goal. In general I think this is fine, but it's clear that the algorithm is not yet sound. Maybe we'll get there one day.
- kayodelycaon 5y ago
  In America, ADA absolutely matters. Algorithms like this can blatantly violate the law and open the company to significant liabilities. I've seen a company repeatedly lose expense lawsuits due to "following process" where the process did not take federal laws protecting workers into account.

WesolyKubeczek 5y ago

Such algorithms usually emphasize biases instead of dissipating them. Sometimes I think though, maybe we should admit in the longer run that some of these biases are actually the truth and all talk of "fair" and "unbiased" is some kind of wishful thinking against all the facts that belongs to the land of teletubbies.

trabant00 5y ago

> wouldn’t algorithmic decisions be preferable to biased human judgement? At this moment? Big NO. Algorithms - are programmed by humans - trained on past data (which by definition includes all the "isms") - have a gajillion corner cases in which they perform hilariously bad - don't understand the data they are processing so they can't reason something must be garbage in How's that Amazon recommendation system working out for you compared to a human recommendation? That's the current state of the art algorithm on a very simple problem.

thebean11 5y ago
I kind of doubt Amazon's algorithm even uses machine learning, it probably assigns a score based on perf metrics (using a human-written algorithm), then fires employees below some cutoff.
thefounder 5y ago
Do you really need an AI to measure the performance(i.e require training as you said) ? I thibk a dumb counter would do the job.
- playpause 5y ago
  Who mentioned AI?
  - thefounder 5y ago
    What is "- trained on past data (which by definition includes all the "isms")" supposed to mean?
mc32 5y ago
>” trained on past data (which by definition includes all the "isms")” Is that necessarily so? Can’t they measure more objective data points like deliveries per hour, missdeliveries, mishandled/broken items (controlled for traffic, pop density, data from deliveries to similar areas, etc) rather than other things like “doesn’t get along with boss:peers, has bad breath, does cat-calls, etc? Now, I admit this has too much Taylorism in it. To much of squeezing the last drop out of people, but I don’t think it has more soft bias.

enumjorge 5y ago

Aside from what others have already replied, I think about how bad automated moderation on big platforms can be, and it doesn’t give me high hopes for this type of software. Think of the stories of an overly aggressive algorithm banning someone for unclear reasons. This has happened to people’s Google accounts. Since Google has limited customer support, people often have a hard time contending the decision. They are locked out of an important slice of their digital lives (email, photos, etc). Can you imagine this happening except it’s your employer who fired you?

maxehmookau 5y ago

You're neither naive or dumb, but this is peak HN. No, it's not better. Algorithms aren't some magical unbiased utopia. They're created by people, with their own biases. They just add a level of separation between the human bias and the person getting fired. Humans are, at least, accountable to other humans.

2pEXgD0fZ5cF 5y ago

> biased human judgement This is the problem, the software is just trained by human decisions. So in many cases they are set up to become even worse. Generally it is bias cast into software. The worst part is that they manage to give the false impression of neutrality. And that doesn't even include all the other problems (cases being ignored/overlooked, false analysis and errors etc.). Also who guarantees that neutral/fair treatment is ever the actual goal?

wyager 5y ago

Your mistake is in assuming uniform treatment is actually the goal. People have mostly publicly given up that pretense, and now there are plenty of explanations for why fairness isn’t enough and we need to actually have a preference for <group> to <do moral thing>. And even that is just a pretense at some level; most of this stuff is actually politics that has little to do with its stated or even first-order-revealed preference.