7 ms·

Disclaimer - I work in the financial services industry. This is nothing new in most financial service machine learning algorithms have to meet these requiremen

Disclaimer - I work in the financial services industry. This is nothing new in most financial service machine learning algorithms have to meet these requirements. This is why why supervised analytics are more popular, and one of the reason why algorithms such as credit scores are typically generated and structured as scorecards with explicit reason codes for each independent characteristic, variable or vector. This is really needed for a number of different reasons. The biggest is that algorithms are increasingly running our world. Even outside of mortgage, have you ever had to fork over the right to pull your credit score for either rent or (in some limited cases) to apply for a job? Not being able to understand: a) Why you were rejected. b) Why you are paying a higher interest rate. c) What you can do to fix that? Housing and employment is fundamentally a question of human rights, not just banking profitability or academic algorithm improvements. Part of the reason why Credit Scoring (which is intrinsically algorithmic decision-making) is because it replaced the "old boys" network that used to dominate. In this world you went to your banker, and the decision to extend credit was dominated by your personal relationship with a banker who might not be the same ethnicity, religion, nationality, etc as you would dominate your ability to purchase housing. The credit score democratized financial access. It can still be used to discriminate. There have been a number of studies that have shown that simply being from a white community tends to increase the number of other people you can lean on in a emergency, and hence make you less likely to default on a loan. From a pure predictability point of view, a bank is at a lower risk of default lending to someone from that community, but that in turn denies financial access to people not in that community, continuing the trend. A big problem in this kind of financial model is that it's relatively easy to accidentally or deliberately find other variables that mirror "protected" vectors. A simplistic example is zip code, where the zip code reflects a area that is predominately ethnic minorities. So it's not cut and dried. It's my PoV that It's not predictability versus red-tape, and the people trying to do unaccountable analytics in this space are (perhaps in-advertedily) perpetuating racism.

deleted 10y ago
[deleted]
sergioisidoro 10y ago
I think the case of zip code based discrimination is even named. It's called redlining, and is being used in machine learning world to describe indirect discrimination based on certain attributes (eg. discriminate people based on their zip code, a zip code with mostly black population for example)
- jacquesm 10y ago
  Interesting aside (but still On Topic): This works for IP addresses as well as zip codes.
- yummyfajitas 10y ago
  But it's also very important to understand why machine learning systems do this. If you take race neutral data, and run virtually any machine learning system on it, you'll get a race neutral output. I.e., if a $40k/year black person in 10001 is equally likely to default as a $40k/year white person in 10002, then the algorithm can be expected to give them equal credit scores. In the event that an algorithm choose black zip codes for extra penalties, it's because the algorithm has deduced that stereotypes are correct - the bank will make more money not lending to blacks at given rates, because something directly correlated with race and not captured by fields like income/past defaults/etc is predictive. I discuss this in greater detail here, taking linear regression as a toy example: https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discriminatory_algorithms.html https://www.chrisstucchio.com/blog/2016/alien_intelligences_... Having consulted for folks doing lending, I'll mention an insanely difficult $100B startup idea here: build an accurate algorithm that exhibits no disparate impact. The amount of money you'll save banks is staggering.
  - InTheArena 10y ago
    True, and then the question becomes, is the stereotypes correct, or is it that the credit score is then propagating a system that enforces this outcome.
    - vkou 10y ago
robrenaud 10y ago
I think there are two problems with requiring explainable models: Explainable models will be more easily gamed, and they are likely to be less accurate. The features in the models themselves will become less useful at their task. They will be gamed. This is roughly along the lines of Campell's law[1], though I've seen other, better explanations that I can't find. What happens when someone is turned down for reason A and B. They go and fix reason A and B, but in the mean time, so have many other credit seekers, diminishing their predictive value. In that time, the modelers create a new, different explainable model, that no longer uses A and B, but somewhat related predictors C and D, which haven't yet been used, and so haven't yet been gamed, which the original seeker does not meet? Explainable models, being a subset of all models, are likely to not contain the most accurate models. I don't anything about the domain of credit scoring (maybe state of the art models are actually small and understandable?), but in speech recognition, for example, models are constantly growing in complexity, their individual decisions are way beyond explainable to anyone in reasonable amount of time, and they are only getting more powerful as they get larger and more complex. In speech, models are already many gigs. In credit scoring, less accurate models mean higher rates, so there is an aggregate loss. [1] https://en.wikipedia.org/wiki/Campbell%27s_law https://en.wikipedia.org/wiki/Campbell%27s_law
- InTheArena 10y ago
  A fair point, but as a society, we have decided that racial discrimination is not a valid mechanism for banks to profit by. That does result in everyone paying a bit more in interest as the risk pool is larger, but a acceptable tradeoff. In terms of gaming, verification is just as important as scoring If the data you have going into to the system is rigged, and income is not being properly validated, bad things will happen.
  - yummyfajitas 10y ago
    As a society we have directed banks to make bad loans to blacks and charge non-blacks extra to make up the difference? I'd be surprised if even 10% of people know this decision was made. Also, what makes it acceptable to engage in this form of surreptitious wealth redistribution on racial lines?
tomaskafka 10y ago
Yep. Or other -isms: > What if [your hiring algorithm] is weeding out women most likely to be pregnant in the next year? https://theoverspill.wordpress.com/2016/07/04/start-up-linkedins-extra-bidders-lgs-mobile-shakeup-google-maps-v-apple-maps-cracking-android-and-more/#link10 https://theoverspill.wordpress.com/2016/07/04/start-up-linke...
- basseq 10y ago
  This is the law of unintended consequences. Data points are correlated to other data points. Unless your hiring algorithm is a random number generator, you could find some sort of "bias" you didn't intend. At least an algorithm is measurable, repeatable, and consistent. Yes, you have to monitor the output of any process for unintended consequences, then you make tweaks as appropriate to eliminate unintended outcomes.
wrp 10y ago
I was trying to think of how to specify this position more precisely: 1. Certain factors are thought to be causally correlated with higher default rates by individuals. 2. Certain populations have higher incidences of individuals matching such factors. 3. These populations have higher than average default rates. 4. Other factors that correlate with membership in these populations will thus be also correlated with higher defalt rates. 5. For an algorithm to be fair, it must be restricted to using only vectors for which causality can be proved. One of the questions is, even if a vector is "proved" to be causal, would it be disallowed if it also strongly predicted membership in a protected population?
DominikR 10y ago
> It's my PoV that It's not predictability versus red-tape, and the people trying to do unaccountable analytics in this space are (perhaps in-advertedily) perpetuating racism. I was with you until that last line. It is not racism to provide better credit conditions to groups that demonstrably have a lower risk of defaulting on a credit. It would be racism if you offer worse conditions to a certain group without any rational business related explanation. If some group has a 2x chance on defaulting on a credit then it isn't about the skin of their color - they get to pay more interest because it is more risky to offer this group a credit in the first place. With that logic you could just as well say that offering better credit conditions to rich people is racist against poor people.
- InTheArena 10y ago
  You said: I was with you until that last line. It is not racism to provide better credit conditions to groups that demonstrably have a lower risk of defaulting on a credit. It absolutely is, under ECOA, which explicitly maintains that you may not discriminate by race for loans. And btw, as someone very familiar with credit scoring once mentioned to me, a small part of the reason the credit scores do not take into account what you currently make is that ability to pay is often a very poor predictor for willingness to pay. this burned a lot of people in 2006/2007 when it was assumed a good credit score justified a big loan, without any proper verification of income, assets and liabilities.
  - DominikR 10y ago
    I didn't say that it is okay to discriminate by race for anything, I'm against that. What I'm saying is that people should be treated fairly instead of using some kind of affirmative action. If you did a proper verification of income, assets, liabilities and education, you'd likely find that some groups would have to receive worse credit conditions than others. This doesn't mean that the system is or would be racist, it would strictly evaluate on socio-economic background, not based on race. What I reject is to handicap one group to make up for the losses that another group generates for a bank by artificially lowering standards for certain groups. Although this is made with best intentions, it is inherently racist (it presumes that some races are inferior to others - hence the introduction of lower standards), whereas evaluating the way I described would not be racist. Btw: I'm myself a second-generation minority. If I'd have had the option to navigate through life with some social security autopilot and affirmative action provided by society I would have never made it to the upper strata of society. You have to understand that expectations for life from where I and many others started were much lower than for most others, so people from this level are much more willing to accept a lower living standard provided for free than working hard for the chance to a higher standard. From the perspective of upper middle class people these kinds of affirmative action and social benefits might seem like genuinely helping people whereas from my perspective it seems like poison that could have had inhibited my will to work on myself and fight to move upwards in society.

deleted 10y ago

[deleted]

sergioisidoro 10y ago

I think the case of zip code based discrimination is even named. It's called redlining, and is being used in machine learning world to describe indirect discrimination based on certain attributes (eg. discriminate people based on their zip code, a zip code with mostly black population for example)

jacquesm 10y ago
Interesting aside (but still On Topic): This works for IP addresses as well as zip codes.
yummyfajitas 10y ago
But it's also very important to understand why machine learning systems do this. If you take race neutral data, and run virtually any machine learning system on it, you'll get a race neutral output. I.e., if a $40k/year black person in 10001 is equally likely to default as a $40k/year white person in 10002, then the algorithm can be expected to give them equal credit scores. In the event that an algorithm choose black zip codes for extra penalties, it's because the algorithm has deduced that stereotypes are correct - the bank will make more money not lending to blacks at given rates, because something directly correlated with race and not captured by fields like income/past defaults/etc is predictive. I discuss this in greater detail here, taking linear regression as a toy example: https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discriminatory_algorithms.html https://www.chrisstucchio.com/blog/2016/alien_intelligences_... Having consulted for folks doing lending, I'll mention an insanely difficult $100B startup idea here: build an accurate algorithm that exhibits no disparate impact. The amount of money you'll save banks is staggering.
- InTheArena 10y ago
  True, and then the question becomes, is the stereotypes correct, or is it that the credit score is then propagating a system that enforces this outcome.
  - vkou 10y ago

robrenaud 10y ago

I think there are two problems with requiring explainable models: Explainable models will be more easily gamed, and they are likely to be less accurate. The features in the models themselves will become less useful at their task. They will be gamed. This is roughly along the lines of Campell's law[1], though I've seen other, better explanations that I can't find. What happens when someone is turned down for reason A and B. They go and fix reason A and B, but in the mean time, so have many other credit seekers, diminishing their predictive value. In that time, the modelers create a new, different explainable model, that no longer uses A and B, but somewhat related predictors C and D, which haven't yet been used, and so haven't yet been gamed, which the original seeker does not meet? Explainable models, being a subset of all models, are likely to not contain the most accurate models. I don't anything about the domain of credit scoring (maybe state of the art models are actually small and understandable?), but in speech recognition, for example, models are constantly growing in complexity, their individual decisions are way beyond explainable to anyone in reasonable amount of time, and they are only getting more powerful as they get larger and more complex. In speech, models are already many gigs. In credit scoring, less accurate models mean higher rates, so there is an aggregate loss. [1] https://en.wikipedia.org/wiki/Campbell%27s_law https://en.wikipedia.org/wiki/Campbell%27s_law

InTheArena 10y ago
A fair point, but as a society, we have decided that racial discrimination is not a valid mechanism for banks to profit by. That does result in everyone paying a bit more in interest as the risk pool is larger, but a acceptable tradeoff. In terms of gaming, verification is just as important as scoring If the data you have going into to the system is rigged, and income is not being properly validated, bad things will happen.
- yummyfajitas 10y ago
  As a society we have directed banks to make bad loans to blacks and charge non-blacks extra to make up the difference? I'd be surprised if even 10% of people know this decision was made. Also, what makes it acceptable to engage in this form of surreptitious wealth redistribution on racial lines?

tomaskafka 10y ago

Yep. Or other -isms: > What if [your hiring algorithm] is weeding out women most likely to be pregnant in the next year? https://theoverspill.wordpress.com/2016/07/04/start-up-linkedins-extra-bidders-lgs-mobile-shakeup-google-maps-v-apple-maps-cracking-android-and-more/#link10 https://theoverspill.wordpress.com/2016/07/04/start-up-linke...

basseq 10y ago
This is the law of unintended consequences. Data points are correlated to other data points. Unless your hiring algorithm is a random number generator, you could find some sort of "bias" you didn't intend. At least an algorithm is measurable, repeatable, and consistent. Yes, you have to monitor the output of any process for unintended consequences, then you make tweaks as appropriate to eliminate unintended outcomes.

wrp 10y ago

I was trying to think of how to specify this position more precisely: 1. Certain factors are thought to be causally correlated with higher default rates by individuals. 2. Certain populations have higher incidences of individuals matching such factors. 3. These populations have higher than average default rates. 4. Other factors that correlate with membership in these populations will thus be also correlated with higher defalt rates. 5. For an algorithm to be fair, it must be restricted to using only vectors for which causality can be proved. One of the questions is, even if a vector is "proved" to be causal, would it be disallowed if it also strongly predicted membership in a protected population?

DominikR 10y ago

> It's my PoV that It's not predictability versus red-tape, and the people trying to do unaccountable analytics in this space are (perhaps in-advertedily) perpetuating racism. I was with you until that last line. It is not racism to provide better credit conditions to groups that demonstrably have a lower risk of defaulting on a credit. It would be racism if you offer worse conditions to a certain group without any rational business related explanation. If some group has a 2x chance on defaulting on a credit then it isn't about the skin of their color - they get to pay more interest because it is more risky to offer this group a credit in the first place. With that logic you could just as well say that offering better credit conditions to rich people is racist against poor people.

InTheArena 10y ago
You said: I was with you until that last line. It is not racism to provide better credit conditions to groups that demonstrably have a lower risk of defaulting on a credit. It absolutely is, under ECOA, which explicitly maintains that you may not discriminate by race for loans. And btw, as someone very familiar with credit scoring once mentioned to me, a small part of the reason the credit scores do not take into account what you currently make is that ability to pay is often a very poor predictor for willingness to pay. this burned a lot of people in 2006/2007 when it was assumed a good credit score justified a big loan, without any proper verification of income, assets and liabilities.
- DominikR 10y ago
  I didn't say that it is okay to discriminate by race for anything, I'm against that. What I'm saying is that people should be treated fairly instead of using some kind of affirmative action. If you did a proper verification of income, assets, liabilities and education, you'd likely find that some groups would have to receive worse credit conditions than others. This doesn't mean that the system is or would be racist, it would strictly evaluate on socio-economic background, not based on race. What I reject is to handicap one group to make up for the losses that another group generates for a bank by artificially lowering standards for certain groups. Although this is made with best intentions, it is inherently racist (it presumes that some races are inferior to others - hence the introduction of lower standards), whereas evaluating the way I described would not be racist. Btw: I'm myself a second-generation minority. If I'd have had the option to navigate through life with some social security autopilot and affirmative action provided by society I would have never made it to the upper strata of society. You have to understand that expectations for life from where I and many others started were much lower than for most others, so people from this level are much more willing to accept a lower living standard provided for free than working hard for the chance to a higher standard. From the perspective of upper middle class people these kinds of affirmative action and social benefits might seem like genuinely helping people whereas from my perspective it seems like poison that could have had inhibited my will to work on myself and fight to move upwards in society.