7 ms·

That is difficult to do because there are many things that correlate with race. So even if you remove race explicitly, things like neighborhoods, level of educa

That is difficult to do because there are many things that correlate with race. So even if you remove race explicitly, things like neighborhoods, level of education and such end up being proxies. Removing race from the model is a very hard problem.

mlyle 6y ago
One example: Credit card issuers have been known to take into account places that you have made purchases in assessing risk. If you shop at businesses that are primarily frequented by minorities, odds are you'll get a lower credit limit than if you didn't.
- mc32 6y ago
  I guess the question is why would that be germane to issuing credit? Is it correlated to something that indicated actual credit risk? If it makes erroneous assumptions correlations, would it not behoove them to eliminate the contamination?
  - mlyle 6y ago
    Sure: some of your purchase history definitely indicates behaviors that indicate you're a credit risk. Some of your purchase history indicates you're likely black (and if you're black, your credit risk is, empirically, a little higher-- even if it is not fair to not issue credit to people because they're black). ML will figure out both but can't explain the hidden variables: it'll figure out behaviors that directly indicate higher risk, and behaviors that indicate you're a minority that tends to be a higher risk.
    - mc32 6y ago
      Is it too entangled to say disregard anything that leads to a path that makes inferences based on demographics? Although I suppose age is also tough to avoid in assigning risk.
      - mlyle 6y ago
        Almost anything you'd use is correlated to demographics somehow or another. We tend to think certain systems are "OK" and not too racist, because they have a well-articulated rationale independent of demographics. But ML is not very good at explaining rationale for decisions, so we don't have the benefit of that argument and it's difficult to even empirically understand how biased the system is. This is a hard problem.
    - unishark 6y ago
      Basing your entire decision on something as simple as the applicant's income will presumably produce outcomes that differ with race. Is that what you mean by racist? Or do you mean the system identifies some individuals as high credit risk when they aren't and may actually lose money versus a more colorblind algorithm?
      - mlyle 6y ago
        What's racist is squishy. Many people would say that identifying people who spend a lot of money at bars or casinos as a credit risk isn't racist, even if it happens to pick up more minorities. The mechanism for the credit risk and behavior seems tightly correlated. Many people would also say that identifying people who spend some money at clothing retailers that market to minorities as a credit risk is racist. Here, the relationship seems like a hidden way to spot minorities, who happen to be more of a credit risk. When banks drew red lines around all the minority neighborhoods and didn't lend to people there, because they couldn't consider race anymore, most thought this behavior unacceptable, even though it did genuinely reduce credit risk for banks. ML can't explain its rationale, and very often does the exact same thing inside an opaque box.
        unishark 6y ago
        Presumably black people who are great loan customers would be incorrectly refused by a system based on fashion or addresses alone, so your answer seems to be that yes, racist methods actually lose the bank money. My tendency is to think that since it is optimized to maximize profit, the more features about a person we get in the data, the less "racist" the system becomes under this definition. Yes it's more possible to "hide" racism in a complex method using tons of features, but increasingly less likely it would happen. If we can use income and debt and whatever other things to make either a good classifier that ignores race, or an inferior one that sneakily identifies race and bases the decision on that, mathematical optimization of the model should result in the former. ML can be trusted more than humans in this situation, not less. Of course there's still the issue of the data being biased, which is where it all started.
        6y ago
    - unishark 6y ago
      By the way many methods are excellent at explaining their decisions. The favorite is decision trees which break the decision up into discrete decisions for individual features. E.g. a person was put in class 1 because their income was above X, their savings was above Y, etc. Other "shallow" methods like regression are fairly easy to interpret as well. It is mainly deep learning for which this is difficult. Which I'm sure is the method in the twitter discussion, but that is a different kind of problem where interpretability doesn't have a clear use.
      - mlyle 6y ago
        When we say "ML", we usually don't mean regression. We not mean deep learning, but at least we mean SVM, SGD, etc. They may be less inscrutable than deep learning, but can easily still draw a box around most of the black people in a clever, non-obvious way.
- lawnchair_larry 6y ago
  Do you have a source?
  - mlyle 6y ago
    e.g. https://consumerist.com/2008/12/22/amex-lowers-your-credit-limit-if-you-shop-where-deadbeats-shop/ https://consumerist.com/2008/12/22/amex-lowers-your-credit-l... Capital One has admitted doing the same, as well as a number of other lenders. There's been analyses done since, that show that when controlled for all other variables in credit reports, credit issuers issue less credit to blacks.. e.g. Cohen-Cole, 2011, Credit Card Redlining, Review of Economics and Statistics
    - lawnchair_larry 6y ago
      Credit Card applications do not ask for race. I suspect it’s more likely based on zip code or similar.
      - mlyle 6y ago
        I said "One example: Credit card issuers have been known to take into account places that you have made purchases in assessing risk. If you shop at businesses that are primarily frequented by minorities, odds are you'll get a lower credit limit than if you didn't" You asked for a source. I provided a source of the same. Yes, things like purchase histories are used as proxies for race and used to deny credit to mostly people of color. Then you go here: > Credit Card applications do not ask for race. which was never asserted. > I suspect it’s more likely based on zip code or similar. Yes, zip code can be another proxy for race.
        lawnchair_larry 6y ago
        Yes, thanks for the example of the former. As for the latter, no, it’s not a proxy for race. It might be highly correlated with race, but that is very different. I don’t know why everyone does these gymnastics to find a racist angle to everything, but it’s nonsense and it’s very harmful to spread that narrative. Nobody is searching for black zipcodes and using that as an input to deny credit. The goal is to increase revenue by giving as much credit as possible with the least risk possible, so it wouldn’t even make sense. Scoring based on where you shop is still behavioral analysis. I think it’s a bad policy, but it isn’t targeting people of color. Your zip code affects insurance rates too, and that’s based on claims. It doesn’t cost more to insure your car in south side chicago than beverly hills because there are black people there, it costs more because there are more claims. The same effect is seen in “white” zip codes that are in areas with severe winters.
lawnchair_larry 6y ago
Or is race the proxy?

mlyle 6y ago

One example: Credit card issuers have been known to take into account places that you have made purchases in assessing risk. If you shop at businesses that are primarily frequented by minorities, odds are you'll get a lower credit limit than if you didn't.

mc32 6y ago
I guess the question is why would that be germane to issuing credit? Is it correlated to something that indicated actual credit risk? If it makes erroneous assumptions correlations, would it not behoove them to eliminate the contamination?
- mlyle 6y ago
  Sure: some of your purchase history definitely indicates behaviors that indicate you're a credit risk. Some of your purchase history indicates you're likely black (and if you're black, your credit risk is, empirically, a little higher-- even if it is not fair to not issue credit to people because they're black). ML will figure out both but can't explain the hidden variables: it'll figure out behaviors that directly indicate higher risk, and behaviors that indicate you're a minority that tends to be a higher risk.
  - mc32 6y ago
    Is it too entangled to say disregard anything that leads to a path that makes inferences based on demographics? Although I suppose age is also tough to avoid in assigning risk.
    - mlyle 6y ago
      Almost anything you'd use is correlated to demographics somehow or another. We tend to think certain systems are "OK" and not too racist, because they have a well-articulated rationale independent of demographics. But ML is not very good at explaining rationale for decisions, so we don't have the benefit of that argument and it's difficult to even empirically understand how biased the system is. This is a hard problem.
  - unishark 6y ago
    Basing your entire decision on something as simple as the applicant's income will presumably produce outcomes that differ with race. Is that what you mean by racist? Or do you mean the system identifies some individuals as high credit risk when they aren't and may actually lose money versus a more colorblind algorithm?
    - mlyle 6y ago
      What's racist is squishy. Many people would say that identifying people who spend a lot of money at bars or casinos as a credit risk isn't racist, even if it happens to pick up more minorities. The mechanism for the credit risk and behavior seems tightly correlated. Many people would also say that identifying people who spend some money at clothing retailers that market to minorities as a credit risk is racist. Here, the relationship seems like a hidden way to spot minorities, who happen to be more of a credit risk. When banks drew red lines around all the minority neighborhoods and didn't lend to people there, because they couldn't consider race anymore, most thought this behavior unacceptable, even though it did genuinely reduce credit risk for banks. ML can't explain its rationale, and very often does the exact same thing inside an opaque box.
      - unishark 6y ago
        Presumably black people who are great loan customers would be incorrectly refused by a system based on fashion or addresses alone, so your answer seems to be that yes, racist methods actually lose the bank money. My tendency is to think that since it is optimized to maximize profit, the more features about a person we get in the data, the less "racist" the system becomes under this definition. Yes it's more possible to "hide" racism in a complex method using tons of features, but increasingly less likely it would happen. If we can use income and debt and whatever other things to make either a good classifier that ignores race, or an inferior one that sneakily identifies race and bases the decision on that, mathematical optimization of the model should result in the former. ML can be trusted more than humans in this situation, not less. Of course there's still the issue of the data being biased, which is where it all started.
        6y ago
  - unishark 6y ago
    By the way many methods are excellent at explaining their decisions. The favorite is decision trees which break the decision up into discrete decisions for individual features. E.g. a person was put in class 1 because their income was above X, their savings was above Y, etc. Other "shallow" methods like regression are fairly easy to interpret as well. It is mainly deep learning for which this is difficult. Which I'm sure is the method in the twitter discussion, but that is a different kind of problem where interpretability doesn't have a clear use.
    - mlyle 6y ago
      When we say "ML", we usually don't mean regression. We not mean deep learning, but at least we mean SVM, SGD, etc. They may be less inscrutable than deep learning, but can easily still draw a box around most of the black people in a clever, non-obvious way.
lawnchair_larry 6y ago
Do you have a source?
- mlyle 6y ago
  e.g. https://consumerist.com/2008/12/22/amex-lowers-your-credit-limit-if-you-shop-where-deadbeats-shop/ https://consumerist.com/2008/12/22/amex-lowers-your-credit-l... Capital One has admitted doing the same, as well as a number of other lenders. There's been analyses done since, that show that when controlled for all other variables in credit reports, credit issuers issue less credit to blacks.. e.g. Cohen-Cole, 2011, Credit Card Redlining, Review of Economics and Statistics
  - lawnchair_larry 6y ago
    Credit Card applications do not ask for race. I suspect it’s more likely based on zip code or similar.
    - mlyle 6y ago
      I said "One example: Credit card issuers have been known to take into account places that you have made purchases in assessing risk. If you shop at businesses that are primarily frequented by minorities, odds are you'll get a lower credit limit than if you didn't" You asked for a source. I provided a source of the same. Yes, things like purchase histories are used as proxies for race and used to deny credit to mostly people of color. Then you go here: > Credit Card applications do not ask for race. which was never asserted. > I suspect it’s more likely based on zip code or similar. Yes, zip code can be another proxy for race.
      - lawnchair_larry 6y ago
        Yes, thanks for the example of the former. As for the latter, no, it’s not a proxy for race. It might be highly correlated with race, but that is very different. I don’t know why everyone does these gymnastics to find a racist angle to everything, but it’s nonsense and it’s very harmful to spread that narrative. Nobody is searching for black zipcodes and using that as an input to deny credit. The goal is to increase revenue by giving as much credit as possible with the least risk possible, so it wouldn’t even make sense. Scoring based on where you shop is still behavioral analysis. I think it’s a bad policy, but it isn’t targeting people of color. Your zip code affects insurance rates too, and that’s based on claims. It doesn’t cost more to insure your car in south side chicago than beverly hills because there are black people there, it costs more because there are more claims. The same effect is seen in “white” zip codes that are in areas with severe winters.

lawnchair_larry 6y ago

Or is race the proxy?