3 ms·

> One way to make the system more ethical, that is less descriminating based on certain features like race or gender, is to often lower the threshold of accepta

by msla 6y ago

> One way to make the system more ethical, that is less descriminating based on certain features like race or gender, is to often lower the threshold of acceptance for the marginalized classes and raise it for the other ones. Or don't input those values at all. Is it even legal for a human to take those things into consideration? Regardless, the AI morally shouldn't, so there's no reason for the data to even be in the input.

cmendel 6y ago
Right; however, the problem is that ML is generally pretty good at finding underlying patterns in data, so even if you make an algorithm race-blind it can still construct a dummy variable that is effectively race. Unfortunately, it does this by codifying pre-existant social outcomes which are often not free of bias.
- RobPfeifer 6y ago
  This is why disparate impact backtesting and go-forward monitoring should be done on all credit model releases
hectormalot 6y ago
As someone who works in this field: the challenge is more nuanced. For example: Even if you don’t input ethnicity as a value, you might input ZIP codes. ZIP codes however can be very predictive of someone’s ethnic background, and you might still end up unfairly discriminating against certain groups of people. The Amazon recruiting AI failure is another great example of this, I think it didn’t take gender into account, but discriminated against activities like ‘head of women’s association’ etc. Finally, what’s ethical is often up for discussion. Depending on context, people will optimize for equality (equal % of credits approved for both genders) vs helping those behind (Higher % approved for those historically disadvantaged) vs helping those that ‘deserve’ it of groups were unknown (finance such that default rates of both groups are expected to be equal, likely benefiting those with already good credit scores). I don’t think any of these answers is wrong, but they are mutually exclusive. One challenge I don’t see discussed that often is that the current system of many humans evaluating loan applications leads to ‘noise‘ in outcomes so that we (on average) don’t see that much bias. (Eg a few people of certain ethnicity not getting loans because a few people are biased against them) A single AI system scoring all applications however is consistent in its bias against a group, leading to more obvious effects (eg most people in a certain area not receiving loans because the algorithm is biased against it) In my opinion, an important part is the right on human review of meaningful automated decisions. Not mentioned that often, but it’s also part of the GDPR.

by msla 6y ago

cmendel 6y ago

Right; however, the problem is that ML is generally pretty good at finding underlying patterns in data, so even if you make an algorithm race-blind it can still construct a dummy variable that is effectively race. Unfortunately, it does this by codifying pre-existant social outcomes which are often not free of bias.

RobPfeifer 6y ago
This is why disparate impact backtesting and go-forward monitoring should be done on all credit model releases

hectormalot 6y ago

As someone who works in this field: the challenge is more nuanced. For example: Even if you don’t input ethnicity as a value, you might input ZIP codes. ZIP codes however can be very predictive of someone’s ethnic background, and you might still end up unfairly discriminating against certain groups of people. The Amazon recruiting AI failure is another great example of this, I think it didn’t take gender into account, but discriminated against activities like ‘head of women’s association’ etc. Finally, what’s ethical is often up for discussion. Depending on context, people will optimize for equality (equal % of credits approved for both genders) vs helping those behind (Higher % approved for those historically disadvantaged) vs helping those that ‘deserve’ it of groups were unknown (finance such that default rates of both groups are expected to be equal, likely benefiting those with already good credit scores). I don’t think any of these answers is wrong, but they are mutually exclusive. One challenge I don’t see discussed that often is that the current system of many humans evaluating loan applications leads to ‘noise‘ in outcomes so that we (on average) don’t see that much bias. (Eg a few people of certain ethnicity not getting loans because a few people are biased against them) A single AI system scoring all applications however is consistent in its bias against a group, leading to more obvious effects (eg most people in a certain area not receiving loans because the algorithm is biased against it) In my opinion, an important part is the right on human review of meaningful automated decisions. Not mentioned that often, but it’s also part of the GDPR.