3 ms·

> We can and should build models that attempt to correct those biases, and we also can and should attempt to de-bias the data itself. What if reality itself is

> We can and should build models that attempt to correct those biases, and we also can and should attempt to de-bias the data itself. What if reality itself is the problem and the source of the "bias"? E.g. say that your ML algo gives less loans to African Americans, but it turns out that in reality African Americans pay back loans less often. Do you accept that some people will have a harder time getting a loan? Or do you instead work to equalise the loan rate across protected characteristics? Or maybe all characteristics?

MontyCarloHall 5y ago
The real answer is that you work to address the underlying reasons that certain demographics are more likely to be delinquent on loans than others. Of course, this would take enormous amounts of effort and decades to happen, which is why nobody wants to hear the real solution to the problem, and instead just want a quick and easy superficial fix (“it must be the algorithm’s fault”).
seoaeu 5y ago
It turns out we actually have laws that answer that question: you cannot determine the loan rate based on protected characteristics. In fact, that's what "protected" means in this context. Laundering the bias through an algorithm doesn't make it any more OK or more legal, it just makes it harder to detect the discrimination.
- commandlinefan 5y ago
  That's the "disparate impact" line of reasoning: if anything results in unequal outcomes, no matter how evenly-handed it was administered, it is racist and must be adjusted until the outcomes are "equitable". It shouldn't be hard (although apparently is for some people) to see what a disaster that line of thinking is.
- hackinthebochs 5y ago
  This doesn't really address the core issue. There's a difference between determining loan rate based on race of the applicant, and using features that correlate with failed loan repayment that happen to overrepresented among black people. While it is illegal to use race as an input to your algorithm, biases in outcomes will still be present given the bias in the relevant non-protected properties. Are ML algorithms required to normalize for race in the face of bias in the relevant variables?

MontyCarloHall 5y ago

The real answer is that you work to address the underlying reasons that certain demographics are more likely to be delinquent on loans than others. Of course, this would take enormous amounts of effort and decades to happen, which is why nobody wants to hear the real solution to the problem, and instead just want a quick and easy superficial fix (“it must be the algorithm’s fault”).

seoaeu 5y ago

It turns out we actually have laws that answer that question: you cannot determine the loan rate based on protected characteristics. In fact, that's what "protected" means in this context. Laundering the bias through an algorithm doesn't make it any more OK or more legal, it just makes it harder to detect the discrimination.

commandlinefan 5y ago
That's the "disparate impact" line of reasoning: if anything results in unequal outcomes, no matter how evenly-handed it was administered, it is racist and must be adjusted until the outcomes are "equitable". It shouldn't be hard (although apparently is for some people) to see what a disaster that line of thinking is.
hackinthebochs 5y ago
This doesn't really address the core issue. There's a difference between determining loan rate based on race of the applicant, and using features that correlate with failed loan repayment that happen to overrepresented among black people. While it is illegal to use race as an input to your algorithm, biases in outcomes will still be present given the bias in the relevant non-protected properties. Are ML algorithms required to normalize for race in the face of bias in the relevant variables?