4 ms·

The loan example comes up a lot but I'm not sure why. We already use machine learning to evaluate credit ratings in the US. It is not always a fair system but i

The loan example comes up a lot but I'm not sure why. We already use machine learning to evaluate credit ratings in the US. It is not always a fair system but it is standard practice and nobody asks why they are turned down for a loan or demands to know how the system works (this information is proprietary and banks would claim their algorithms are a trade secret since a better algorithm gives them an edge on pricing loans). >>You'd want the reason your application was not granted to be something reasonable (like a poor credit history) and not something like "the particular combination of inputs triggered some weird path and rejected you offhand." This is a low dimensional bias. If there is increased risk of default from high A & B & C & D but not high A or high B or high C or high D, then the combination or parameters is what matters even if it is not easy to explain. Typically in a high dimensional space most of the volume is far from the axes so it is unlikely that things will line up along some preconceived set of inputs. As it is 'poor credit history' is in fact an index that amalgamates a large number of different parameters so I'm not sure if that really explains why the loan was rejected or simply gives a simple name for a complicated thing. In general yes, it is good to thoroughly debug any ML algorithm and make sure that is is doing roughy what you think that it is doing. A lot of times this process can be quite complicated and relies on a lot of intuition & heuristics. While thoroughly testing a ML solution is certainly best practice, I'm not sure if having a highly skilled researcher conducting an in-depth mathematical analysis of an algorithm would really make it 'interpretable'.

entee 6y ago
I’m pretty sure that despite the fact that statistical models are used in credit scores, those models are subject to regulatory review. For example, I’m pretty sure a “black, therefore, no loan” algorithm is illegal, regardless of whether that parameter was learned or programmed. It’s a huge problem in some of these models that latent racism can bubble up into a putatively unbiased algorithm. In some industries, including finance, I think that can expose to some regulatory risk, hence explainability being quite helpful.
xeRTRex 6y ago
I had experience developing automated underwriting models in the insurance industry. As the models are becoming more sophisticated and adapting machine learning, heavier scrutiny is coming from regulators. For good reasons, guardrails are required to not only protect against discrimination, but also so proxies can't be used either. For example, we can't include race in health and life underwriting decisions. But since zip code is highly predictive of race, that attribute must also be excluded. I'm not familiar with banking regulations, but imagine similar policies are applied. In these cases, being able to demonstrate that a model isn't discriminating is not only ethically important, but in many cases is legally required.
- disgruntledphd2 6y ago
  This comment is very, very true. While you as a person may not be able to get details of the algorithms and methods used, the regulators get exhaustive documentation about every aspect of them. With GDPR, explainability is going to become a requirement, but it will probably take some test cases before this happens.
- snovv_crash 6y ago
  It's more than discrimination in banking, you need to show that there are no side channels where insider information might be fed in.
stdbrouw 6y ago
> This is a low dimensional bias. If there is increased risk of default from high A & B & C & D but not high A or high B or high C or high D, then the combination or parameters is what matters even if it is not easy to explain. However, even if we expect that a non-obvious combination of parameters will matter, we usually expect the hyperplane of our predictions to be at least a little bit smooth in various ways: monotonic or curved instead of jagged, small changes in input should cause only small changes in output, etc. Not just to make it easier to understand, also because the kinds of processes we study tend to behave that way. For regions of high density, machine learning does exactly what you say it does: generate high-quality predictions or categorizations, even if the particular path that led it there is nonobvious or weird. But these models are generally not sensitive to how they categorize or predict unusual combinations of inputs and as to predictive quality for those edge cases, all bets are off. A very simple case is polynomial regression, which can be tuned to perfectly fit the training data but outside of the training set might oscillate wildly or go to infinity -- and this isn't really the result of overfitting, it's just what polynomials do.
bkandel 6y ago
The loan example comes up because of a long history of explicit discrimination in the US against Black borrowers (https://en.wikipedia.org/wiki/Redlining https://en.wikipedia.org/wiki/Redlining). Because of that history, people are justifiably skeptical when someone says: "I know that this industry discriminated against you in the past, but trust me, this new and completely opaque system is TOTALLY not going to discriminate against you anymore." Especially when the new systems have demonstrable racial bias (https://www.realtor.com/news/trends/black-communities-higher-mortgage-rates/ https://www.realtor.com/news/trends/black-communities-higher... among many many other sources).

entee 6y ago

I’m pretty sure that despite the fact that statistical models are used in credit scores, those models are subject to regulatory review. For example, I’m pretty sure a “black, therefore, no loan” algorithm is illegal, regardless of whether that parameter was learned or programmed. It’s a huge problem in some of these models that latent racism can bubble up into a putatively unbiased algorithm. In some industries, including finance, I think that can expose to some regulatory risk, hence explainability being quite helpful.

xeRTRex 6y ago

I had experience developing automated underwriting models in the insurance industry. As the models are becoming more sophisticated and adapting machine learning, heavier scrutiny is coming from regulators. For good reasons, guardrails are required to not only protect against discrimination, but also so proxies can't be used either. For example, we can't include race in health and life underwriting decisions. But since zip code is highly predictive of race, that attribute must also be excluded. I'm not familiar with banking regulations, but imagine similar policies are applied. In these cases, being able to demonstrate that a model isn't discriminating is not only ethically important, but in many cases is legally required.

disgruntledphd2 6y ago
This comment is very, very true. While you as a person may not be able to get details of the algorithms and methods used, the regulators get exhaustive documentation about every aspect of them. With GDPR, explainability is going to become a requirement, but it will probably take some test cases before this happens.
snovv_crash 6y ago
It's more than discrimination in banking, you need to show that there are no side channels where insider information might be fed in.

stdbrouw 6y ago

> This is a low dimensional bias. If there is increased risk of default from high A & B & C & D but not high A or high B or high C or high D, then the combination or parameters is what matters even if it is not easy to explain. However, even if we expect that a non-obvious combination of parameters will matter, we usually expect the hyperplane of our predictions to be at least a little bit smooth in various ways: monotonic or curved instead of jagged, small changes in input should cause only small changes in output, etc. Not just to make it easier to understand, also because the kinds of processes we study tend to behave that way. For regions of high density, machine learning does exactly what you say it does: generate high-quality predictions or categorizations, even if the particular path that led it there is nonobvious or weird. But these models are generally not sensitive to how they categorize or predict unusual combinations of inputs and as to predictive quality for those edge cases, all bets are off. A very simple case is polynomial regression, which can be tuned to perfectly fit the training data but outside of the training set might oscillate wildly or go to infinity -- and this isn't really the result of overfitting, it's just what polynomials do.

bkandel 6y ago

The loan example comes up because of a long history of explicit discrimination in the US against Black borrowers (https://en.wikipedia.org/wiki/Redlining https://en.wikipedia.org/wiki/Redlining). Because of that history, people are justifiably skeptical when someone says: "I know that this industry discriminated against you in the past, but trust me, this new and completely opaque system is TOTALLY not going to discriminate against you anymore." Especially when the new systems have demonstrable racial bias (https://www.realtor.com/news/trends/black-communities-higher-mortgage-rates/ https://www.realtor.com/news/trends/black-communities-higher... among many many other sources).