4 ms·

Disallowing the use of race as a signal in machine learning / ad targeting could help intercultural awareness. Here's the last paragraph of the article: """ O

by jchrisa 10y ago

Disallowing the use of race as a signal in machine learning / ad targeting could help intercultural awareness. Here's the last paragraph of the article: """ O’Neil also proposes updating existing civic rights oriented legislation to make it clear that it encompasses computerized algorithms. One thing that her book has already made quite clear – far from being coolly scientific, Big Data comes with all the biases of its creators. It’s time to stop pretending that people wielding the most numbers necessarily have the right answers.

ben_jones 10y ago
Conversely, what if the numbers are unbias and we are in fact the bias ones. What if, for example, Irish Americans are actually more prone to purchase Guinness beer because we feel that it has a slightly higher alcohol content then all those other beers? Tongue and cheek response, but "wielding numbers"? Politicians don't misuse statistics to get what they want, people misuse statistics to get what we want. We shouldn't ban statistics and we shouldn't ignore the fact that both "sides" are equally susceptible.
JanneVee 10y ago
> Disallowing the use of race as a signal in machine learning / ad targeting could help intercultural awareness. Uhm, racial segregation is still a thing. Even if the signal of race is gone the geography of where a person lives could be signal enough to discriminate.
nl 10y ago
Disallowing the use of race as a signal in machine learning That sounds like a great idea, but it isn't as simple as it appears. Most systems won't have "race" directly encoded as a feature, but that is insufficient. See slide 22 from [1], further in depth discussion discussion from [2] or just this feature (which many systems could automatically discover): Feature6578 = Loc=EastOakland && Income<10k I don't know what the solution to this is, but it's a pretty hard problem, and even the best intention are insufficient. Delip proposes in [1] to introduce a "fairness constraint" where the probability of a favourable outcome in the majority class is the same as in a protected class. This sounds sensible, but make optimising systems much harder. [1] http://deliprao.com/archives/129 http://deliprao.com/archives/129 [2] https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discriminatory_algorithms.html https://www.chrisstucchio.com/blog/2016/alien_intelligences_...
- mseebach 10y ago
  The problem is deciding which features are reasonable to include. Yes, location and income can probably predict race to a certain precision, and if you're building a race-detector-by-proxy, you'd want to include those. But what if you're evaluating eligibility for a mortgage? Certainly location and income are critical. A black person in East Oakland with an income below $10k is going to struggle with a mortgage, but so is a white person with those characteristics (and a black person in Connecticut making $250k is going to do fine and so is a white person). But if more black people than white legitimately are in a situation where they might struggle to service a mortgage, more black people than white would get flagged as such by an objective algorithm. Does that make the algorithm racist? The slide seems to imply so (although without speaker-notes it's difficult to ascertain). The flip-side is that under a human-biased system, the first white person might get the mortgage based on being white, and the second black person might be denied. That would be reversed under an algorithmic system only considering legitimate factors, and surely that's progress?
  - nl 10y ago
    There isn't a simple answer here - like in most non-trivial data science/ML questions. The only think I think worth pointing out is that in the mortgage example the current location of a person isn't as major an influence as income. The fairness test here seems somewhat reasonable: a person on $10K/y who lives in Oakland should probably have the same outcome as someone on $10K/y from some very white area - all other things being equal. The flip-side is that under a human-biased system, the first white person might get the mortgage based on being white, and the second black person might be denied. That would be reversed under an algorithmic system only considering legitimate factors, and surely that's progress? Yes, I think so? Or at least it can be audited (in theory at least - although see sentencing guideline systems for the problems with this).
  - yummyfajitas 10y ago
    The problem is actually different. Race is often very predictive. Blacks don't perform worse simply because they have income < $10k. Blacks perform worse on many measures even if you hold all that stuff equal. Here are a few of the many studies reproducing this: http://ftp.iza.org/dp8733.pdf http://ftp.iza.org/dp8733.pdf http://www.mindingthecampus.org/2010/09/the_underperformance_problem/ http://www.mindingthecampus.org/2010/09/the_underperformance... https://randomcriticalanalysis.wordpress.com/2015/05/16/on-concentrated-poverty-and-its-effects-on-academic-outcomes/ https://randomcriticalanalysis.wordpress.com/2015/05/16/on-c... https://randomcriticalanalysis.wordpress.com/2015/11/22/on-the-relationship-between-negative-home-owner-equity-and-racial-demographics/#my_analysis https://randomcriticalanalysis.wordpress.com/2015/11/22/on-t... If you were right, then the machine learning algorithm would quickly determine that race doesn't matter. Redundant encoding, direct encoding, etc would be irrelevant - the algorithm would ignore it as noise. The problem is that the algorithm is supposed to uncover hidden patterns that predict loan default. But then there is one specific hidden pattern that is socially and legally taboo to detect, yet also highly predictive. (Incidentally, if you can solve this problem in lending, it's easily a unicorn startup.)
denzil_correa 10y ago
> Disallowing the use of race as a signal in machine learning / ad targeting could help intercultural awareness Many a time, algorithms infer race from secondary or tertiary variables. This problem is compounded by two more factors - (1) Unable to explain WHY an algorithm has reached a decision aka interpretability and (2) Focus of algorithm building to maximize accuracy regardless of other valid costs. There is some pretty interesting work going on this area. http://fairness.haverford.edu/ http://fairness.haverford.edu/

by jchrisa 10y ago

ben_jones 10y ago

Conversely, what if the numbers are unbias and we are in fact the bias ones. What if, for example, Irish Americans are actually more prone to purchase Guinness beer because we feel that it has a slightly higher alcohol content then all those other beers? Tongue and cheek response, but "wielding numbers"? Politicians don't misuse statistics to get what they want, people misuse statistics to get what we want. We shouldn't ban statistics and we shouldn't ignore the fact that both "sides" are equally susceptible.

JanneVee 10y ago

> Disallowing the use of race as a signal in machine learning / ad targeting could help intercultural awareness. Uhm, racial segregation is still a thing. Even if the signal of race is gone the geography of where a person lives could be signal enough to discriminate.

nl 10y ago

Disallowing the use of race as a signal in machine learning That sounds like a great idea, but it isn't as simple as it appears. Most systems won't have "race" directly encoded as a feature, but that is insufficient. See slide 22 from [1], further in depth discussion discussion from [2] or just this feature (which many systems could automatically discover): Feature6578 = Loc=EastOakland && Income<10k I don't know what the solution to this is, but it's a pretty hard problem, and even the best intention are insufficient. Delip proposes in [1] to introduce a "fairness constraint" where the probability of a favourable outcome in the majority class is the same as in a protected class. This sounds sensible, but make optimising systems much harder. [1] http://deliprao.com/archives/129 http://deliprao.com/archives/129 [2] https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discriminatory_algorithms.html https://www.chrisstucchio.com/blog/2016/alien_intelligences_...

mseebach 10y ago
The problem is deciding which features are reasonable to include. Yes, location and income can probably predict race to a certain precision, and if you're building a race-detector-by-proxy, you'd want to include those. But what if you're evaluating eligibility for a mortgage? Certainly location and income are critical. A black person in East Oakland with an income below $10k is going to struggle with a mortgage, but so is a white person with those characteristics (and a black person in Connecticut making $250k is going to do fine and so is a white person). But if more black people than white legitimately are in a situation where they might struggle to service a mortgage, more black people than white would get flagged as such by an objective algorithm. Does that make the algorithm racist? The slide seems to imply so (although without speaker-notes it's difficult to ascertain). The flip-side is that under a human-biased system, the first white person might get the mortgage based on being white, and the second black person might be denied. That would be reversed under an algorithmic system only considering legitimate factors, and surely that's progress?
- nl 10y ago
  There isn't a simple answer here - like in most non-trivial data science/ML questions. The only think I think worth pointing out is that in the mortgage example the current location of a person isn't as major an influence as income. The fairness test here seems somewhat reasonable: a person on $10K/y who lives in Oakland should probably have the same outcome as someone on $10K/y from some very white area - all other things being equal. The flip-side is that under a human-biased system, the first white person might get the mortgage based on being white, and the second black person might be denied. That would be reversed under an algorithmic system only considering legitimate factors, and surely that's progress? Yes, I think so? Or at least it can be audited (in theory at least - although see sentencing guideline systems for the problems with this).
- yummyfajitas 10y ago
  The problem is actually different. Race is often very predictive. Blacks don't perform worse simply because they have income < $10k. Blacks perform worse on many measures even if you hold all that stuff equal. Here are a few of the many studies reproducing this: http://ftp.iza.org/dp8733.pdf http://ftp.iza.org/dp8733.pdf http://www.mindingthecampus.org/2010/09/the_underperformance_problem/ http://www.mindingthecampus.org/2010/09/the_underperformance... https://randomcriticalanalysis.wordpress.com/2015/05/16/on-concentrated-poverty-and-its-effects-on-academic-outcomes/ https://randomcriticalanalysis.wordpress.com/2015/05/16/on-c... https://randomcriticalanalysis.wordpress.com/2015/11/22/on-the-relationship-between-negative-home-owner-equity-and-racial-demographics/#my_analysis https://randomcriticalanalysis.wordpress.com/2015/11/22/on-t... If you were right, then the machine learning algorithm would quickly determine that race doesn't matter. Redundant encoding, direct encoding, etc would be irrelevant - the algorithm would ignore it as noise. The problem is that the algorithm is supposed to uncover hidden patterns that predict loan default. But then there is one specific hidden pattern that is socially and legally taboo to detect, yet also highly predictive. (Incidentally, if you can solve this problem in lending, it's easily a unicorn startup.)

denzil_correa 10y ago

> Disallowing the use of race as a signal in machine learning / ad targeting could help intercultural awareness Many a time, algorithms infer race from secondary or tertiary variables. This problem is compounded by two more factors - (1) Unable to explain WHY an algorithm has reached a decision aka interpretability and (2) Focus of algorithm building to maximize accuracy regardless of other valid costs. There is some pretty interesting work going on this area. http://fairness.haverford.edu/ http://fairness.haverford.edu/