4 ms·

I suppose we can always weasel around with the definition of "why". Why was my claim refused? Because you have a large number of consonants in your surname wh

I suppose we can always weasel around with the definition of "why". Why was my claim refused? Because you have a large number of consonants in your surname which, when combined with the fact your phone number has a prime number of "4" digits, leads to an increase risk of fraud. Why do those factors indicate fraud? Because our training data indicates that they do. Why does your training data indicate that? ...

PeterisP 10y ago
Well, that's not a problem - when you say "my training data indicates that they do" what you really mean is "our previous experience shows that applicants with these factors have a significantly elevated chance of not paying back our money", which by itself is a valid reason as it is objective and based on real world facts. A bigger problem would be when your features actually turn out to be proxies for 'undesired' features. For example, if you do put names in as features, then any credit risk system will learn that e.g. "Tyrone Washington" is a higher risk customer (assuming everything else is equal) than "Eric Shmidt", since those names are highly predictive of most socioeconomic factors; however, you'll risk a judge saying that this implements racial discrimination even if you exclude race directly, since those names are even more predictive of skin color than income.
- fixermark 10y ago
  Kind of, mostly. If the training data is correlating consonants in surname or prime number of '4' digits in a telephone number with increased credit risk, there's a really solid case there to ask where the training data sample came from and check it for biased sourcing.
  - PeterisP 10y ago
    If the system is correlating consonants in surname or prime number of '4' digits in a telephone number with increased risk, well, then in practice that simply means that you have a bad system with a serious overfitting problem - i.e., not a problem with your data but a problem with your learning process that's obtained "superstition" by treating random noise as important signal. However, if you do some reason analysis, then this is useful and gives a lot of interesting information. Names will be correlated with socioeconomic status and ethnicity, and also with education level [of your parents], income, etc. Addresses obviously are strong indicators especially if you combine them with data about that location or even particular building if that's available. Even phone numbers can carry a lot of true information e.g. location, or for countries that handed out new ranges for mobile phones, it would be correlated with how long you've had your phone number/how often you change them, which is a proxy for stability. I'd bet that you could even build a somewhat predictive model of credit risk based on character-by-character analysis of email adresses - even excluding domain names, the fact if someone chose fluffybunny420 or jonathan.cumbersnatch carries some signal; not very much, but if you don't have better data then it will serve as a vague proxy for age, lifestyle and frivolousness (and in case of domain names, possibly employer) which all are important influencers of risk.
    - fixermark 10y ago
      This explanation is solid and speaks loudly to the benefits of right to explanation. Because if the explanation is "We've found signal in the data that correlates strongly with social biases (example: surname analysis---of course there's increased risk, if the lender is operating in a country where people don't trust anyone whose name starts with "Mc", for example), and we can use that signal to basically ethnically profile without having to admit that we're ethnically profiling..." that's bad, and the law may address it. "Redlining" practices aren't bad because they don't economically work. They do economically work; if one demographic is societally disadvantaged, you don't do your firm any short-run favors by catering to them, because they aren't where the money is. They're bad because they work by unfairly shifting the pie away from people who don't actually deserve to have the pie shifted away from them---it's a variant on "sins of the fathers" reasoning to bias on categories like surname, or race, or demographics that correlate strongly with those demographics (like geography, in cities in the United States). There are moral reasons to not allow those practices to get codified into algorithms so the people using them can excuse bad behavior as "just following the computer's orders."
- crististm 10y ago
  Maybe Eric Shmidt is a lower risk customer because he's opinion on "don't be evil" is that is a bad idea? :)
- matt4077 10y ago
  If others in this thread are correct and the explanation will be a sort of sensitivity analysis (meaning that the factors relevant for the decision have to be communicated) that will amount to discrimination. It doesn't matter if some employee judges customers by their skin colour or if someone build a convenient algorithm to do it for them. I think name-based decisions are pretty obviously racist. I'm wondering about the less correlated variables such as addresses.
  - PeterisP 10y ago
    I'm under impression that USA is still segregated enough so that for many places, an address (or even districts/towns) will be a very strong indicator of race.
    - throwawaysocks 10y ago
      Definitely.

PeterisP 10y ago

Well, that's not a problem - when you say "my training data indicates that they do" what you really mean is "our previous experience shows that applicants with these factors have a significantly elevated chance of not paying back our money", which by itself is a valid reason as it is objective and based on real world facts. A bigger problem would be when your features actually turn out to be proxies for 'undesired' features. For example, if you do put names in as features, then any credit risk system will learn that e.g. "Tyrone Washington" is a higher risk customer (assuming everything else is equal) than "Eric Shmidt", since those names are highly predictive of most socioeconomic factors; however, you'll risk a judge saying that this implements racial discrimination even if you exclude race directly, since those names are even more predictive of skin color than income.

fixermark 10y ago
Kind of, mostly. If the training data is correlating consonants in surname or prime number of '4' digits in a telephone number with increased credit risk, there's a really solid case there to ask where the training data sample came from and check it for biased sourcing.
- PeterisP 10y ago
  If the system is correlating consonants in surname or prime number of '4' digits in a telephone number with increased risk, well, then in practice that simply means that you have a bad system with a serious overfitting problem - i.e., not a problem with your data but a problem with your learning process that's obtained "superstition" by treating random noise as important signal. However, if you do some reason analysis, then this is useful and gives a lot of interesting information. Names will be correlated with socioeconomic status and ethnicity, and also with education level [of your parents], income, etc. Addresses obviously are strong indicators especially if you combine them with data about that location or even particular building if that's available. Even phone numbers can carry a lot of true information e.g. location, or for countries that handed out new ranges for mobile phones, it would be correlated with how long you've had your phone number/how often you change them, which is a proxy for stability. I'd bet that you could even build a somewhat predictive model of credit risk based on character-by-character analysis of email adresses - even excluding domain names, the fact if someone chose fluffybunny420 or jonathan.cumbersnatch carries some signal; not very much, but if you don't have better data then it will serve as a vague proxy for age, lifestyle and frivolousness (and in case of domain names, possibly employer) which all are important influencers of risk.
  - fixermark 10y ago
    This explanation is solid and speaks loudly to the benefits of right to explanation. Because if the explanation is "We've found signal in the data that correlates strongly with social biases (example: surname analysis---of course there's increased risk, if the lender is operating in a country where people don't trust anyone whose name starts with "Mc", for example), and we can use that signal to basically ethnically profile without having to admit that we're ethnically profiling..." that's bad, and the law may address it. "Redlining" practices aren't bad because they don't economically work. They do economically work; if one demographic is societally disadvantaged, you don't do your firm any short-run favors by catering to them, because they aren't where the money is. They're bad because they work by unfairly shifting the pie away from people who don't actually deserve to have the pie shifted away from them---it's a variant on "sins of the fathers" reasoning to bias on categories like surname, or race, or demographics that correlate strongly with those demographics (like geography, in cities in the United States). There are moral reasons to not allow those practices to get codified into algorithms so the people using them can excuse bad behavior as "just following the computer's orders."
crististm 10y ago
Maybe Eric Shmidt is a lower risk customer because he's opinion on "don't be evil" is that is a bad idea? :)
matt4077 10y ago
If others in this thread are correct and the explanation will be a sort of sensitivity analysis (meaning that the factors relevant for the decision have to be communicated) that will amount to discrimination. It doesn't matter if some employee judges customers by their skin colour or if someone build a convenient algorithm to do it for them. I think name-based decisions are pretty obviously racist. I'm wondering about the less correlated variables such as addresses.
- PeterisP 10y ago
  I'm under impression that USA is still segregated enough so that for many places, an address (or even districts/towns) will be a very strong indicator of race.
  - throwawaysocks 10y ago
    Definitely.