3 ms·
Disallowing the use of race as a signal in machine learning That sounds like a great idea, but it isn't as simple as it appears. Most systems won't have "race"
by nl 10y ago
Disallowing the use of race as a signal in machine learning
That sounds like a great idea, but it isn't as simple as it appears. Most systems won't have "race" directly encoded as a feature, but that is insufficient.
See slide 22 from [1], further in depth discussion discussion from [2] or just this feature (which many systems could automatically discover):
Feature6578 = Loc=EastOakland && Income<10k
I don't know what the solution to this is, but it's a pretty hard problem, and even the best intention are insufficient.
Delip proposes in [1] to introduce a "fairness constraint" where the probability of a favourable outcome in the majority class is the same as in a protected class. This sounds sensible, but make optimising systems much harder.
[1] http://deliprao.com/archives/129 http://deliprao.com/archives/129
[2] https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discriminatory_algorithms.html https://www.chrisstucchio.com/blog/2016/alien_intelligences_...
- mseebach 10y agoThe problem is deciding which features are reasonable to include. Yes, location and income can probably predict race to a certain precision, and if you're building a race-detector-by-proxy, you'd want to include those. But what if you're evaluating eligibility for a mortgage? Certainly location and income are critical. A black person in East Oakland with an income below $10k is going to struggle with a mortgage, but so is a white person with those characteristics (and a black person in Connecticut making $250k is going to do fine and so is a white person). But if more black people than white legitimately are in a situation where they might struggle to service a mortgage, more black people than white would get flagged as such by an objective algorithm. Does that make the algorithm racist? The slide seems to imply so (although without speaker-notes it's difficult to ascertain). The flip-side is that under a human-biased system, the first white person might get the mortgage based on being white, and the second black person might be denied. That would be reversed under an algorithmic system only considering legitimate factors, and surely that's progress?
- nl 10y agoThere isn't a simple answer here - like in most non-trivial data science/ML questions. The only think I think worth pointing out is that in the mortgage example the current location of a person isn't as major an influence as income. The fairness test here seems somewhat reasonable: a person on $10K/y who lives in Oakland should probably have the same outcome as someone on $10K/y from some very white area - all other things being equal. The flip-side is that under a human-biased system, the first white person might get the mortgage based on being white, and the second black person might be denied. That would be reversed under an algorithmic system only considering legitimate factors, and surely that's progress? Yes, I think so? Or at least it can be audited (in theory at least - although see sentencing guideline systems for the problems with this).
- yummyfajitas 10y agoThe problem is actually different. Race is often very predictive. Blacks don't perform worse simply because they have income < $10k. Blacks perform worse on many measures even if you hold all that stuff equal. Here are a few of the many studies reproducing this: http://ftp.iza.org/dp8733.pdf http://ftp.iza.org/dp8733.pdf http://www.mindingthecampus.org/2010/09/the_underperformance_problem/ http://www.mindingthecampus.org/2010/09/the_underperformance... https://randomcriticalanalysis.wordpress.com/2015/05/16/on-concentrated-poverty-and-its-effects-on-academic-outcomes/ https://randomcriticalanalysis.wordpress.com/2015/05/16/on-c... https://randomcriticalanalysis.wordpress.com/2015/11/22/on-the-relationship-between-negative-home-owner-equity-and-racial-demographics/#my_analysis https://randomcriticalanalysis.wordpress.com/2015/11/22/on-t... If you were right, then the machine learning algorithm would quickly determine that race doesn't matter. Redundant encoding, direct encoding, etc would be irrelevant - the algorithm would ignore it as noise. The problem is that the algorithm is supposed to uncover hidden patterns that predict loan default. But then there is one specific hidden pattern that is socially and legally taboo to detect, yet also highly predictive. (Incidentally, if you can solve this problem in lending, it's easily a unicorn startup.)