4 ms·

This is where things get complex with ML. Any dimension that is correlated with the protected categories can potentially get you into a biased and discriminator

This is where things get complex with ML. Any dimension that is correlated with the protected categories can potentially get you into a biased and discriminatory outcome. Because embedding are not necessarily transparent, it’s hard to evaluate what the model attaches to. The most accessible example is anything to do with fine grained location, such as zip+4. Because people tend to cluster, zip+4 has been identified as a close proxy for race and other protected classes. When we are talking ML-derived decisions and the nuanced approach, we have to be cognizant of the fact that certain repayment patterns and behaviors could also be strongly correlated to a protected class.

darawk 8y ago
What if the correct predictor really is the zip code, though? Let's say there's a city that's predominantly black, and it has a statistically significantly higher default rate than other cities. However, the default rate in that city is high for residents of all races who live there. In this scenario, the 'true' predictor really is the city, but it also correlated with race. Is simple correlation to a protected class sufficient to nix a variable?
- sologoub 8y ago
  The protected classes are not protected because the data is somehow statistically not significant or cannot be a predictor. The classes are protected because as a society, we have decided that greater social good is achieved by sacrificing these predictors. The question is then what is it that you are after with this prediction? Max profit or max social good with enough profit.
  - darawk 8y ago
    My point is that lots of things correlate with race. Almost any conceivable variable you could devise to predict loan defaults correlates with race. Gender and age are maybe the only two things I can think of that almost certainly don't correlate with race (although African Americans do tend to die earlier so...age might be out). Given that, what standard can you realistically use here?

darawk 8y ago

What if the correct predictor really is the zip code, though? Let's say there's a city that's predominantly black, and it has a statistically significantly higher default rate than other cities. However, the default rate in that city is high for residents of all races who live there. In this scenario, the 'true' predictor really is the city, but it also correlated with race. Is simple correlation to a protected class sufficient to nix a variable?

sologoub 8y ago
The protected classes are not protected because the data is somehow statistically not significant or cannot be a predictor. The classes are protected because as a society, we have decided that greater social good is achieved by sacrificing these predictors. The question is then what is it that you are after with this prediction? Max profit or max social good with enough profit.
- darawk 8y ago
  My point is that lots of things correlate with race. Almost any conceivable variable you could devise to predict loan defaults correlates with race. Gender and age are maybe the only two things I can think of that almost certainly don't correlate with race (although African Americans do tend to die earlier so...age might be out). Given that, what standard can you realistically use here?