3 ms·

Where things go wrong is when race comes into play to begin with. In your example it is clear that credit scores - or whatever equivalent is used in the country

Where things go wrong is when race comes into play to begin with. In your example it is clear that credit scores - or whatever equivalent is used in the country in question - are a factor in deciding whether an applicant should be allowed to take out a certain loan. That check should be done on base of that applicant's credit score irrespective of where he lives, what car he drives, what his name is, what colour his skin has or any other unrelated characteristic. While there might be relationships between average credit score and the aforementioned data points they are not in itself indicative of the likelihood of the applicant defaulting on the loan. The credit score is a better one, as is their income for the last X months and other similar data. It might well be the case that a totally 'colour-blind' scoring algorithm still ends up sorting people by one of those mentioned, unrelated data points. This is not racism, it is just the consequence of the mentioned relationships. Calling it racism is a sign of ignorance and does a disservice to the effort to get rid of true racism.

deogeo 8y ago
The problem is, e.g. banks will want to use all the data they can to make a decision on what sort of loan to offer you. Employment history, travel history, family medical history, as much data on your spending habits as they can get, how much money your relatives have, which properties you own and are they likely to rise or drop in value, where, when, and how you drive (which affects likelihood of having a car accident -> unexpected expenses or death -> affects loan repayment), which websites you visit, how much time you spend on your phone, in bars, etc. etc.. Each has a plausible justification for inclusion. But the more data points you have, the better you're able to predict race from them (or anything else you're not supposed to). So how do you decide if an algorithm is discriminatory? Maybe a whitelist or blacklist of what data it can use?
- im3w1l 8y ago
  I think we can use dangerous data in safe ways. If you use how you drive only to predict likelihood of car accident that's ok. If you use it to predict future wage growth it's not ok. If you use it in a blackbox model it's not ok. If you use time in bar to predict cirrhosis it's ok. etc
  - axlprose 8y ago
    >If you use how you drive only to predict likelihood of car accident that's ok. We may say that now, but what if later some algorithm finds there's a hidden correlation between driving performance and race/gender? will it still be an ok metric then? where do we draw the line?
    - im3w1l 8y ago
      I said that expecting such a correlation to exist. The line is between inferences of bad behavior -> bad result such as "you drive risky so you must be at risk of an accident" and pure correlation "you drive risky so you must be a man so you must be at increased risk of prostate cancer". And conclusions should be assumed to be of the latter kind unless shown otherwise.

deogeo 8y ago

The problem is, e.g. banks will want to use all the data they can to make a decision on what sort of loan to offer you. Employment history, travel history, family medical history, as much data on your spending habits as they can get, how much money your relatives have, which properties you own and are they likely to rise or drop in value, where, when, and how you drive (which affects likelihood of having a car accident -> unexpected expenses or death -> affects loan repayment), which websites you visit, how much time you spend on your phone, in bars, etc. etc.. Each has a plausible justification for inclusion. But the more data points you have, the better you're able to predict race from them (or anything else you're not supposed to). So how do you decide if an algorithm is discriminatory? Maybe a whitelist or blacklist of what data it can use?

im3w1l 8y ago
I think we can use dangerous data in safe ways. If you use how you drive only to predict likelihood of car accident that's ok. If you use it to predict future wage growth it's not ok. If you use it in a blackbox model it's not ok. If you use time in bar to predict cirrhosis it's ok. etc
- axlprose 8y ago
  >If you use how you drive only to predict likelihood of car accident that's ok. We may say that now, but what if later some algorithm finds there's a hidden correlation between driving performance and race/gender? will it still be an ok metric then? where do we draw the line?
  - im3w1l 8y ago
    I said that expecting such a correlation to exist. The line is between inferences of bad behavior -> bad result such as "you drive risky so you must be at risk of an accident" and pure correlation "you drive risky so you must be a man so you must be at increased risk of prostate cancer". And conclusions should be assumed to be of the latter kind unless shown otherwise.