5 ms·

Notice how the article jumps from: "these algorithms are trained on historical data, and poor uneducated people (often racial minorities) have a historical tre

Notice how the article jumps from: "these algorithms are trained on historical data, and poor uneducated people (often racial minorities) have a historical trend of being more likely to succumb to predatory loan advertisements" to "Even if this is the most common question being asked on Google [..] this shows that algorithms do in fact encode our prejudice" It's hard to make the case for prejudice when the example involves historical data, so the author weasels in an unrelated example to make the case that there is prejudice. So which is it? There are correlates to credit worthiness which are indeed illegal, allegedly to correct bias. But the legal arguments have moved away from the concept of bias and now squarely focus on impact. See the recent Supreme Court decision on disparate impact for instance. This seems to be the author's point of view to: "Why is this ignorant? Because of the well-known fact that removing explicit racial features from data does not eliminate an algorithm’s ability to learn race. If racial features disproportionately correlate with crime (as they do in the US), then an algorithm which learns race is actually doing exactly what it is designed to do!" So for the author, anything that affects one ethnic group differently is racist, whether or not it is born by the data. Prejudice or bias are irrelevant to his definition. But then why focus on race? No scoring algorithm is perfect, and some people are always going to be shortchanged. So why not just declare by fiat: let's ignore people's height, it's height-ist. Height correlates to income and to credit worthiness. Even if you remove height from the data, your algorithm will end up picking correlates which can become a proxy to height... and the same will be true with virtually any attribute you can think of. So why race?

theseatoms 11y ago
There are two separate issues being discussed at once: 1) baseless prejudice, expressed en masse, and therefore gaining algorithmic notability, and 2) empirically backed, statistically defensible stereotypes. This second issue seems harder to resolve, because it will require lawmakers to clearly articulate a "bright line" law in the language of the algorithm itself. As the article acknowledges, algorithms can "learn" race, even if race is not an explicitly defined explanatory variable. Looking forward to part two. Edit: Why race? Because it's an obvious and easy example. I believe the author would be on board with your height example.
- murbard2 11y ago
  Regarding the second issue, Griggs v. Duke Power Co. asserted that requiring a high-school degree for a job that arguably didn't strictly require one was discriminatory because it could act as a proxy to race. If the author is on board with height, I can take him down all the way to a reductio where you give everyone the exact same credit terms.
  - theseatoms 11y ago
    Well hopefully that case doesn't hold too much power via legal precedent, which is a concept I never fully understood. I can very much appreciate your reductio argument, but if i had to wager, that's the way we're headed. Seems like aspiring lawyers should start cracking open the ML textbooks, because that case is a pandora's box.
- gohrt 11y ago
  > statistically defensible stereotypes. "defensible" is a weasel word. It goes back to the old "correlation vs causation" Machine learning choose the "shape" of a decision surface. Optimizing parameters of that surface may be defensible, or may nor be. Your algorithm may learn that people with my skin color, or who live in my neighborhood, on have a certain average behavior on your objective function. I would word your #1 and #2 as (1) baseless prejudice that midjudges people in general, (2) statistical prejudice that misjudges many individuals. But it's still a leap to treat me as the "average" in that group, instead of judging me on the content of my character. What guided your choice of parameters? Even if you did the best within practical limits, it still may be unfair.
  - theseatoms 11y ago
    Point taken. Though, for better or worse, correlation can very frequently meaningfully improve the decision-making process, whether the organization employing said algorithm is lending to consumers or something else.
stcredzero 11y ago
Indeed. The roots of the words prejudice are well known to most English speakers. It's supposed to denote "pre" judgement. Judgement before actual evidence. Regarding impact: It could well be argued that current-day impact is the result of prejudice operating in the past. The interaction of society, economics, and demographics are complex. Hopefully, we'll just muddle through, somehow.
- tpllaha 11y ago
  Right... But isn't "judgement before actual evidence" all ML algorithms do? They try to statistically evaluate something about a set's members based on other features, because actual evidence of what we want to find out is missing. In my opinion this means that we either: 1) ignore the problem during the algorithm design phase and analyze the output with caution, or 2) decide that any statistical classification of human beings is "unfair" or even illegal (because it will have to discriminate people based on some properties that we know about them - be it race, or height or whatever) I don't see any middle-ground, since "prejudice" in my opinion is an unavoidable effect of ML algorithms as we know them.
  - myrryr 11y ago
    In many respects all ML does is evaluate all the evidence before it. It cuts both ways.
- gohrt 11y ago
  Evidence is not binary. "Prejudice" can be "premature judice". > Hopefully, we'll just muddle through, somehow. This is not a good enough answer for the people who are actually being hurt, now. For these people, it's not an abstract problem.
j2kun 11y ago
A good definition of fairness should not depend on the semantic meaning of any feature. What features we want to enforce algorithms to be fair wrt is a policy question, not a mathematical one. Mathematically you're right, I shouldn't focus on race. But race is the motivation for the research h, and I think it's a good one.
- msellout 11y ago
  What are your thoughts on Zest Finance (http://www.zestfinance.com http://www.zestfinance.com) ?
- murbard2 11y ago
  I think "unfair" implies prejudice and bias. Do you take it as axiomatic that a system that produces different outcome for different racial groups is "unfair"?
  - j2kun 11y ago
    This is precisely the question :) we just don't know what the right definitions are, and neither do we have a good understanding of the consequences of existing attempts.
    - murbard2 11y ago
      You don't seem to be asking that question at all. You seem to be including disparate outcome in the set of "unfair" things, by fiat, and from there looking for the logical closure of that concept under machine learning. That closure is "Harrison Bergeron"... but one man's modus ponens is another's modus tollens.
      - j2kun 11y ago
        I think I'm quite clear: we don't have an accepted and useful mathematical definition of what fairness means. This should, at least on some level, line up with our society's legal standards (as the corresponding mathematical definitions of privacy, security, etc. do) while allowing us to prove theorems or at least make well-defined conjectures about the properties of algorithms. The question is what counts as fair. I'm not saying disparate outcomes is necessary or sufficient, though it receives a lot of attention. Even if we could quantify which disparate outcomes are adverse, it's not obvious how to design algorithms that meet those criteria.
denzil_correa 11y ago
> So for the author, anything that affects one ethnic group differently is racist, whether or not it is born by the data. I would encourage to check out some of the papers at the FAT-ML workshop which was co-located with ICML. I don't think researchers are making that assumption. In addition, race was used as an example because it is one of the sensitive attributes. The decision on what makes an attribute sensitive is based on deep discussions by society. The author of the blog post wants to communicate two points - (1) Algorithms can discriminate and (2) There is a need to understand HOW algorithms discriminate.
- moron4hire 11y ago
  These are givens. You don't create any sort of decision algorithm unless you are trying to ask the computer to discriminate. The question is, do you understand what the political perception of the eventual outcome will be? As an overly simplistic example: someone might point out that your algorithm's negative impact applies 75% of the time to racial minorities. There will be a political shit storm if that is the end of the story. But if you can show that the segmentation is actually 99.8% people in prison, you basically prove your algorithm is politically acceptable and pass the buck of outrage on somewhere else.
empath75 11y ago
So imagine a situation where due to actual prejudice (for example, redlining, segregated, inferior schools, employment discrimination), outcomes and circumstances are clearly predictable by race. If you're not careful with your algorithms, your algorithm and see the world 'as it is' and see it 'as it must be' or 'as it should be'.
netcan 11y ago
I understand you train of thought and it makes sense. If you remove the human element and you are dealing with facts, how can this be described as "racist." On the other hand, would you be comfortable just feeding an algorithm (say one for scoring university applicants) with "race" and allowing it to find (if it is statistically relevant) the races most likely to succeed and fail? I agree that this stuff is hard to nail down in a logically consistent way. But, this stuff has a history. Group discrimination has had horrible results in the past. This is our sloppy duct tape fix. Algorithm based selection is now challenging the assumptions. If an admissions officer offers places only to handsome, white men from upper class backgrounds, we see that as unfair because these things have no direct impact on their value as students. He's an ass. For the sake of argument, imagine an algorithm arrives at the same conclusion as the ass. Our moral instinct is confused. Societal impact is similar. There certainly is something dehumanizing about selection algorithms making decisions. I hate buying insurance or being assessed for loans. It feels bad. Basically, I don't think you can reductio ad absurdum this argument away. There are inconsistencies in our moral sentiments.
- murbard2 11y ago
  If that were the best I could realistically do, yes (though that itself isn't realistic, there are many other criteria). Any other technique would, by definition end up unfavorably affecting a greater number of people. Any imperfect criterion will always exist at the expense of some group of people (tautologically, the group of people to which the criterion is unfavorable). What difference does it make, ethically, if the group happens to be identifiable?
  - netcan 11y ago
    In the mandatory insurance example, its a zero sum game. If an insurance company had no data, they would have to give everyone the average premium. Egality of ignorance.
- yummyfajitas 11y ago
  Our moral instinct is confused because our political agitators have based normative (should) conclusions on positive (is) claims. I.e., "racial stereotypes are incorrect, and therefore it's morally wrong to base decisions on them". See, for example, this long thread where this conflation is made: https://news.ycombinator.com/item?id=8368876 https://news.ycombinator.com/item?id=8368876 The claim is logically false - you can't prove a normative claim without a normative premise. In my example above, one possible normative hypothesis would be "and it's morally wrong to use incorrect premises to make decisions" (or possibly "it's morally wrong to use incorrect racial premises to make decisions"). Now our algorithms are revealing that underlying positive claim (e.g., "racial stereotypes are incorrect") might not be true either. This is confusing, and rightly so - it demonstrates our entire belief system was predicated on nonsense. Hopefully we'll actually respond with some useful analytical philosophy.

theseatoms 11y ago

There are two separate issues being discussed at once: 1) baseless prejudice, expressed en masse, and therefore gaining algorithmic notability, and 2) empirically backed, statistically defensible stereotypes. This second issue seems harder to resolve, because it will require lawmakers to clearly articulate a "bright line" law in the language of the algorithm itself. As the article acknowledges, algorithms can "learn" race, even if race is not an explicitly defined explanatory variable. Looking forward to part two. Edit: Why race? Because it's an obvious and easy example. I believe the author would be on board with your height example.

murbard2 11y ago
Regarding the second issue, Griggs v. Duke Power Co. asserted that requiring a high-school degree for a job that arguably didn't strictly require one was discriminatory because it could act as a proxy to race. If the author is on board with height, I can take him down all the way to a reductio where you give everyone the exact same credit terms.
- theseatoms 11y ago
  Well hopefully that case doesn't hold too much power via legal precedent, which is a concept I never fully understood. I can very much appreciate your reductio argument, but if i had to wager, that's the way we're headed. Seems like aspiring lawyers should start cracking open the ML textbooks, because that case is a pandora's box.
gohrt 11y ago
> statistically defensible stereotypes. "defensible" is a weasel word. It goes back to the old "correlation vs causation" Machine learning choose the "shape" of a decision surface. Optimizing parameters of that surface may be defensible, or may nor be. Your algorithm may learn that people with my skin color, or who live in my neighborhood, on have a certain average behavior on your objective function. I would word your #1 and #2 as (1) baseless prejudice that midjudges people in general, (2) statistical prejudice that misjudges many individuals. But it's still a leap to treat me as the "average" in that group, instead of judging me on the content of my character. What guided your choice of parameters? Even if you did the best within practical limits, it still may be unfair.
- theseatoms 11y ago
  Point taken. Though, for better or worse, correlation can very frequently meaningfully improve the decision-making process, whether the organization employing said algorithm is lending to consumers or something else.

stcredzero 11y ago

Indeed. The roots of the words prejudice are well known to most English speakers. It's supposed to denote "pre" judgement. Judgement before actual evidence. Regarding impact: It could well be argued that current-day impact is the result of prejudice operating in the past. The interaction of society, economics, and demographics are complex. Hopefully, we'll just muddle through, somehow.

tpllaha 11y ago
Right... But isn't "judgement before actual evidence" all ML algorithms do? They try to statistically evaluate something about a set's members based on other features, because actual evidence of what we want to find out is missing. In my opinion this means that we either: 1) ignore the problem during the algorithm design phase and analyze the output with caution, or 2) decide that any statistical classification of human beings is "unfair" or even illegal (because it will have to discriminate people based on some properties that we know about them - be it race, or height or whatever) I don't see any middle-ground, since "prejudice" in my opinion is an unavoidable effect of ML algorithms as we know them.
- myrryr 11y ago
  In many respects all ML does is evaluate all the evidence before it. It cuts both ways.
gohrt 11y ago
Evidence is not binary. "Prejudice" can be "premature judice". > Hopefully, we'll just muddle through, somehow. This is not a good enough answer for the people who are actually being hurt, now. For these people, it's not an abstract problem.

j2kun 11y ago

A good definition of fairness should not depend on the semantic meaning of any feature. What features we want to enforce algorithms to be fair wrt is a policy question, not a mathematical one. Mathematically you're right, I shouldn't focus on race. But race is the motivation for the research h, and I think it's a good one.

msellout 11y ago
What are your thoughts on Zest Finance (http://www.zestfinance.com http://www.zestfinance.com) ?
murbard2 11y ago
I think "unfair" implies prejudice and bias. Do you take it as axiomatic that a system that produces different outcome for different racial groups is "unfair"?
- j2kun 11y ago
  This is precisely the question :) we just don't know what the right definitions are, and neither do we have a good understanding of the consequences of existing attempts.
  - murbard2 11y ago
    You don't seem to be asking that question at all. You seem to be including disparate outcome in the set of "unfair" things, by fiat, and from there looking for the logical closure of that concept under machine learning. That closure is "Harrison Bergeron"... but one man's modus ponens is another's modus tollens.
    - j2kun 11y ago
      I think I'm quite clear: we don't have an accepted and useful mathematical definition of what fairness means. This should, at least on some level, line up with our society's legal standards (as the corresponding mathematical definitions of privacy, security, etc. do) while allowing us to prove theorems or at least make well-defined conjectures about the properties of algorithms. The question is what counts as fair. I'm not saying disparate outcomes is necessary or sufficient, though it receives a lot of attention. Even if we could quantify which disparate outcomes are adverse, it's not obvious how to design algorithms that meet those criteria.

denzil_correa 11y ago

> So for the author, anything that affects one ethnic group differently is racist, whether or not it is born by the data. I would encourage to check out some of the papers at the FAT-ML workshop which was co-located with ICML. I don't think researchers are making that assumption. In addition, race was used as an example because it is one of the sensitive attributes. The decision on what makes an attribute sensitive is based on deep discussions by society. The author of the blog post wants to communicate two points - (1) Algorithms can discriminate and (2) There is a need to understand HOW algorithms discriminate.

moron4hire 11y ago
These are givens. You don't create any sort of decision algorithm unless you are trying to ask the computer to discriminate. The question is, do you understand what the political perception of the eventual outcome will be? As an overly simplistic example: someone might point out that your algorithm's negative impact applies 75% of the time to racial minorities. There will be a political shit storm if that is the end of the story. But if you can show that the segmentation is actually 99.8% people in prison, you basically prove your algorithm is politically acceptable and pass the buck of outrage on somewhere else.

empath75 11y ago

So imagine a situation where due to actual prejudice (for example, redlining, segregated, inferior schools, employment discrimination), outcomes and circumstances are clearly predictable by race. If you're not careful with your algorithms, your algorithm and see the world 'as it is' and see it 'as it must be' or 'as it should be'.

netcan 11y ago

I understand you train of thought and it makes sense. If you remove the human element and you are dealing with facts, how can this be described as "racist." On the other hand, would you be comfortable just feeding an algorithm (say one for scoring university applicants) with "race" and allowing it to find (if it is statistically relevant) the races most likely to succeed and fail? I agree that this stuff is hard to nail down in a logically consistent way. But, this stuff has a history. Group discrimination has had horrible results in the past. This is our sloppy duct tape fix. Algorithm based selection is now challenging the assumptions. If an admissions officer offers places only to handsome, white men from upper class backgrounds, we see that as unfair because these things have no direct impact on their value as students. He's an ass. For the sake of argument, imagine an algorithm arrives at the same conclusion as the ass. Our moral instinct is confused. Societal impact is similar. There certainly is something dehumanizing about selection algorithms making decisions. I hate buying insurance or being assessed for loans. It feels bad. Basically, I don't think you can reductio ad absurdum this argument away. There are inconsistencies in our moral sentiments.

murbard2 11y ago
If that were the best I could realistically do, yes (though that itself isn't realistic, there are many other criteria). Any other technique would, by definition end up unfavorably affecting a greater number of people. Any imperfect criterion will always exist at the expense of some group of people (tautologically, the group of people to which the criterion is unfavorable). What difference does it make, ethically, if the group happens to be identifiable?
- netcan 11y ago
  In the mandatory insurance example, its a zero sum game. If an insurance company had no data, they would have to give everyone the average premium. Egality of ignorance.
yummyfajitas 11y ago
Our moral instinct is confused because our political agitators have based normative (should) conclusions on positive (is) claims. I.e., "racial stereotypes are incorrect, and therefore it's morally wrong to base decisions on them". See, for example, this long thread where this conflation is made: https://news.ycombinator.com/item?id=8368876 https://news.ycombinator.com/item?id=8368876 The claim is logically false - you can't prove a normative claim without a normative premise. In my example above, one possible normative hypothesis would be "and it's morally wrong to use incorrect premises to make decisions" (or possibly "it's morally wrong to use incorrect racial premises to make decisions"). Now our algorithms are revealing that underlying positive claim (e.g., "racial stereotypes are incorrect") might not be true either. This is confusing, and rightly so - it demonstrates our entire belief system was predicated on nonsense. Hopefully we'll actually respond with some useful analytical philosophy.