4 ms·

The problem is deciding which features are reasonable to include. Yes, location and income can probably predict race to a certain precision, and if you're build

The problem is deciding which features are reasonable to include. Yes, location and income can probably predict race to a certain precision, and if you're building a race-detector-by-proxy, you'd want to include those. But what if you're evaluating eligibility for a mortgage? Certainly location and income are critical. A black person in East Oakland with an income below $10k is going to struggle with a mortgage, but so is a white person with those characteristics (and a black person in Connecticut making $250k is going to do fine and so is a white person). But if more black people than white legitimately are in a situation where they might struggle to service a mortgage, more black people than white would get flagged as such by an objective algorithm. Does that make the algorithm racist? The slide seems to imply so (although without speaker-notes it's difficult to ascertain). The flip-side is that under a human-biased system, the first white person might get the mortgage based on being white, and the second black person might be denied. That would be reversed under an algorithmic system only considering legitimate factors, and surely that's progress?

nl 10y ago
There isn't a simple answer here - like in most non-trivial data science/ML questions. The only think I think worth pointing out is that in the mortgage example the current location of a person isn't as major an influence as income. The fairness test here seems somewhat reasonable: a person on $10K/y who lives in Oakland should probably have the same outcome as someone on $10K/y from some very white area - all other things being equal. The flip-side is that under a human-biased system, the first white person might get the mortgage based on being white, and the second black person might be denied. That would be reversed under an algorithmic system only considering legitimate factors, and surely that's progress? Yes, I think so? Or at least it can be audited (in theory at least - although see sentencing guideline systems for the problems with this).
yummyfajitas 10y ago
The problem is actually different. Race is often very predictive. Blacks don't perform worse simply because they have income < $10k. Blacks perform worse on many measures even if you hold all that stuff equal. Here are a few of the many studies reproducing this: http://ftp.iza.org/dp8733.pdf http://ftp.iza.org/dp8733.pdf http://www.mindingthecampus.org/2010/09/the_underperformance_problem/ http://www.mindingthecampus.org/2010/09/the_underperformance... https://randomcriticalanalysis.wordpress.com/2015/05/16/on-concentrated-poverty-and-its-effects-on-academic-outcomes/ https://randomcriticalanalysis.wordpress.com/2015/05/16/on-c... https://randomcriticalanalysis.wordpress.com/2015/11/22/on-the-relationship-between-negative-home-owner-equity-and-racial-demographics/#my_analysis https://randomcriticalanalysis.wordpress.com/2015/11/22/on-t... If you were right, then the machine learning algorithm would quickly determine that race doesn't matter. Redundant encoding, direct encoding, etc would be irrelevant - the algorithm would ignore it as noise. The problem is that the algorithm is supposed to uncover hidden patterns that predict loan default. But then there is one specific hidden pattern that is socially and legally taboo to detect, yet also highly predictive. (Incidentally, if you can solve this problem in lending, it's easily a unicorn startup.)

nl 10y ago

There isn't a simple answer here - like in most non-trivial data science/ML questions. The only think I think worth pointing out is that in the mortgage example the current location of a person isn't as major an influence as income. The fairness test here seems somewhat reasonable: a person on $10K/y who lives in Oakland should probably have the same outcome as someone on $10K/y from some very white area - all other things being equal. The flip-side is that under a human-biased system, the first white person might get the mortgage based on being white, and the second black person might be denied. That would be reversed under an algorithmic system only considering legitimate factors, and surely that's progress? Yes, I think so? Or at least it can be audited (in theory at least - although see sentencing guideline systems for the problems with this).

yummyfajitas 10y ago

The problem is actually different. Race is often very predictive. Blacks don't perform worse simply because they have income < $10k. Blacks perform worse on many measures even if you hold all that stuff equal. Here are a few of the many studies reproducing this: http://ftp.iza.org/dp8733.pdf http://ftp.iza.org/dp8733.pdf http://www.mindingthecampus.org/2010/09/the_underperformance_problem/ http://www.mindingthecampus.org/2010/09/the_underperformance... https://randomcriticalanalysis.wordpress.com/2015/05/16/on-concentrated-poverty-and-its-effects-on-academic-outcomes/ https://randomcriticalanalysis.wordpress.com/2015/05/16/on-c... https://randomcriticalanalysis.wordpress.com/2015/11/22/on-the-relationship-between-negative-home-owner-equity-and-racial-demographics/#my_analysis https://randomcriticalanalysis.wordpress.com/2015/11/22/on-t... If you were right, then the machine learning algorithm would quickly determine that race doesn't matter. Redundant encoding, direct encoding, etc would be irrelevant - the algorithm would ignore it as noise. The problem is that the algorithm is supposed to uncover hidden patterns that predict loan default. But then there is one specific hidden pattern that is socially and legally taboo to detect, yet also highly predictive. (Incidentally, if you can solve this problem in lending, it's easily a unicorn startup.)