5 ms·

This is simply not true. If we train a mortgage engine on redlining data, it will only reproduce the red line if it's predictive. I.e., the algorithm will only

This is simply not true. If we train a mortgage engine on redlining data, it will only reproduce the red line if it's predictive. I.e., the algorithm will only reproduce the red line if the red line is statistically valid. I discuss this in detail here, with numerical examples: https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discriminatory_algorithms.html https://www.chrisstucchio.com/blog/2016/alien_intelligences_... The fact is that machine learning takes biased inputs and produces unbiased outputs, and this is a pretty normal occurrence. If I tell you that my algorithm to target advertisements treated `displayedInterest x isMobile` with 15% more weight than `displayedInterest x isDesktop` because it corrected for bias induced by latency on mobile connections, you'd think nothing of it.

lmm 10y ago
> This is simply not true. If we train a mortgage engine on redlining data, it will only reproduce the red line if it's predictive. I.e., the algorithm will only reproduce the red line if the red line is statistically valid. If you train the algorithm to make the same decisions humans did (which is reasonably common when the humans are considered to be effective experts), it will make the same decisions humans did. If you're comparing outcomes then you'll do better, but even then bias can affect outcomes - if members of group A was sold higher-interest mortgages than equivalent members of group B, then your dataset will show that group B has higher default rates and your algorithm will learn this, even though the difference was only there due to human bias.
- yummyfajitas 10y ago
  But you don't train the algorithm to make the same decision humans did. You train it to maximize your objective function. I algorithmically trade the stock market. I don't train my model by asking whether it trades the way I would. The whole point is for it to do a better job than me! I train the model to maximize my returns [1] in backtests and simulations - if it trades differently than I do, so much the better! [1] More precisely, volatility adjusted returns, and I also penalize model complexity to avoid overfitting.
  - lmm 10y ago
    > But you don't train the algorithm to make the same decision humans did. You train it to maximize your objective function. In the idea case you would, sure. In practice mortgages take 25 years to deliver outcome data and the point isn't necessarily to get better outcomes, the mortgage provider would be content to get exactly the same outcomes (or even slightly worse outcomes) if it let them replace a large number of human employees with an automated system. In most fields of human endeavour you can't backtest, because you don't have access to the outcomes of the decisions you didn't make. And the data often aren't as clear-cut or objective as market prices. The stock market is great but it's atypical of modern "big data" in many ways.
    - yummyfajitas 10y ago
      In most fields of human endeavor you do have an objective function better than "do what humans do". Mortgages are very specifically an area where you do. First of all, there is historical data. Second of all, you can backtest well before 25 years. A couple of weeks ago I wrote a blog post explaining specifically how to make measurements in the presence of delayed reactions - I'm discussing a situation involving sensor networks, literally the same mathematics would work for mortgage default or refinance: https://www.chrisstucchio.com/blog/2016/delayed_reactions.html https://www.chrisstucchio.com/blog/2016/delayed_reactions.ht... Third, mortgage lenders can often backtest alternate decisions because there are pretty straightforward relationships between decisions. Some of them are even mechanistic, e.g. refinance_risk(interest_rate) and default_prob(interest_rate) are monotonically increasing. You seem to think we are living in the exact specific dark age necessary to make your morality play poignant and relevant. That's about as silly as Star Trek landing on all sorts of alien planets, each one designed to highlight one specific social issue from USA 1966.
      - lmm 10y ago
        There's no historical data on the default rates of people who didn't get mortgages, because they didn't get mortgages. I'm willing to believe that theoretical solutions exist. I know from direct personal experience in the big data/lending industry that they are not always applied. If you are claiming that real-world lenders never train their models on human decisions then you are simply wrong. Nice zinger - I hope you weren't saving it for too long. But I'm not going to get into witticisms or personal arguments.
SamBam 10y ago
> The fact is that machine learning takes biased inputs and produces unbiased outputs That is not necessarily true, because it assumes that the biased inputs have no influence on the outputs. Suppose the training data is a bank that has historically given higher-interest loans to minorities. Those loans will default more often, and so the algorithm could "correctly" conclude that minorities default more often. Cause and effect are very hard for an algorithm to distinguish when it's the clustered input variables that are linked together for some opaque reason. The fact is, both statements in this example would be true: "higher interest is correlated with more defaults" and "minority borrowers are correlated with defaults." The algorithm doesn't know one from the other. Sure, more training data could help the algorithm distinguish between the two, but there may not be enough training data that shows the inverse -- low-interest loans given to minorities -- if the historical data is biased enough.
- yummyfajitas 10y ago
  Suppose the training data is a bank that has historically given higher-interest loans to minorities. Those loans will default more often, and so the algorithm could "correctly" conclude that minorities default more often. But if the relationship is actually `defaultProb = a x interest_rate + b`, then the algorithm will reflect that. I discuss this case explicitly in the blog post I linked to in the section "What if black people don't perform as well?", and explicitly give an example of a linear model NOT doing what you claim it will do. The fact is, both statements in this example would be true: "higher interest is correlated with more defaults" and "minority borrowers are correlated with defaults." The algorithm doesn't know one from the other. No, it wouldn't because you'd also have cases of white people with high interest who defaulted. The model using race as a predictive variable would fail to fit those data points. I explicitly discuss this case in my blog post. You are correct that in some cases you don't have enough training data. But in these cases, there is no particular reason for bias to have a particular sign. Again, as I discuss in the blog post I linked, why would Captain Kirk be biased against "black on left guy" vs "black on right guy"? Insufficient training data causes variance, not bias, and could just as easily result in bias in favor of blacks. (In fact, I link to a number of examples where it does.)
  - 10y ago
sharemywin 10y ago
red-line - "The term refers to the presumed practice of mortgage lenders of drawing red lines around portions of a map to indicate areas or neighborhoods in which they do not want to make loans. Redlining on a racial basis has been held by the courts to be an illegal practice." What's interesting is if at some point in future regulators extend "red line" to mean variable highly correlated to a protected class. Because if your using a variable and not checking how it correlates to a protected class and making financial based decisions on it. Couldn't you in affect transform the decision into a graph "women" on on side "men" on another(or what ever your favorite protected class is).
thaw13579 10y ago
I appreciate this analysis, but it's important to note that not everyone wants unbiased model output. Historically, redlining was intentional, and it took laws to stop it. A major risk of automating mortgage engines is that redlining could easily be reinstated (in the form of model parameters) in a way that is both hard to detect and easily deniable.
- yummyfajitas 10y ago
  There are two possible reasons for redlining: 1) I hate black people more than I like money, but blacks do pay back their loans, so I'll redline even if it costs me money. 2) I like money and am neutral towards blacks, but blacks are deadbeats so redlining gets me more money. (There are also 2 other cases. "I love/am neutral towards blacks and they pay back their loans", which results in no disparate impact. Also "I hate blacks and they don't pay back their loans" which is equivalent to (2).) Statistical algorithms solve (1), yet somehow disparate impact isn't fixed. (2) is unspeakable to the modern left - only racists could say that - yet statistics continually reproduces conclusion (2). So obviously the problem must be with the algorithms.
  - dragonwriter 10y ago
    There is also "I am neutral to blacks and view them as, ceteris paribus, equally likely to pay back loans, but white people on average are not neutral toward them, and particularly are not neutral to living in close proximity to them, and the decrease in property values when black people are allowed in otherwise white neighborhoods makes both their default and the default of white people in the neighborhood more likely, and makes deficiency more likely in the event of default of either." (Statistical models that properly account track all-factors likelihood of default on a particular loan do not solve this.)
    - yummyfajitas 10y ago
      True, a model which predicts reality will in fact predict reality. I've never seen any evidence that your hypothesis is true, or even a model that would account for it, but I'm hardly an expert in the field. Assuming this is more than speculation/theoretical nitpicking, do you have citations on the effects you describe? If your theory is more than just idle speculation, this is a great way for non-racists to engage in house price arbitrage insofar as they wish to consume housing. Do you know of any neighborhoods in NYC where I can exploit this arbitrage?

lmm 10y ago

> This is simply not true. If we train a mortgage engine on redlining data, it will only reproduce the red line if it's predictive. I.e., the algorithm will only reproduce the red line if the red line is statistically valid. If you train the algorithm to make the same decisions humans did (which is reasonably common when the humans are considered to be effective experts), it will make the same decisions humans did. If you're comparing outcomes then you'll do better, but even then bias can affect outcomes - if members of group A was sold higher-interest mortgages than equivalent members of group B, then your dataset will show that group B has higher default rates and your algorithm will learn this, even though the difference was only there due to human bias.

yummyfajitas 10y ago
But you don't train the algorithm to make the same decision humans did. You train it to maximize your objective function. I algorithmically trade the stock market. I don't train my model by asking whether it trades the way I would. The whole point is for it to do a better job than me! I train the model to maximize my returns [1] in backtests and simulations - if it trades differently than I do, so much the better! [1] More precisely, volatility adjusted returns, and I also penalize model complexity to avoid overfitting.
- lmm 10y ago
  > But you don't train the algorithm to make the same decision humans did. You train it to maximize your objective function. In the idea case you would, sure. In practice mortgages take 25 years to deliver outcome data and the point isn't necessarily to get better outcomes, the mortgage provider would be content to get exactly the same outcomes (or even slightly worse outcomes) if it let them replace a large number of human employees with an automated system. In most fields of human endeavour you can't backtest, because you don't have access to the outcomes of the decisions you didn't make. And the data often aren't as clear-cut or objective as market prices. The stock market is great but it's atypical of modern "big data" in many ways.
  - yummyfajitas 10y ago
    In most fields of human endeavor you do have an objective function better than "do what humans do". Mortgages are very specifically an area where you do. First of all, there is historical data. Second of all, you can backtest well before 25 years. A couple of weeks ago I wrote a blog post explaining specifically how to make measurements in the presence of delayed reactions - I'm discussing a situation involving sensor networks, literally the same mathematics would work for mortgage default or refinance: https://www.chrisstucchio.com/blog/2016/delayed_reactions.html https://www.chrisstucchio.com/blog/2016/delayed_reactions.ht... Third, mortgage lenders can often backtest alternate decisions because there are pretty straightforward relationships between decisions. Some of them are even mechanistic, e.g. refinance_risk(interest_rate) and default_prob(interest_rate) are monotonically increasing. You seem to think we are living in the exact specific dark age necessary to make your morality play poignant and relevant. That's about as silly as Star Trek landing on all sorts of alien planets, each one designed to highlight one specific social issue from USA 1966.
    - lmm 10y ago
      There's no historical data on the default rates of people who didn't get mortgages, because they didn't get mortgages. I'm willing to believe that theoretical solutions exist. I know from direct personal experience in the big data/lending industry that they are not always applied. If you are claiming that real-world lenders never train their models on human decisions then you are simply wrong. Nice zinger - I hope you weren't saving it for too long. But I'm not going to get into witticisms or personal arguments.

SamBam 10y ago

> The fact is that machine learning takes biased inputs and produces unbiased outputs That is not necessarily true, because it assumes that the biased inputs have no influence on the outputs. Suppose the training data is a bank that has historically given higher-interest loans to minorities. Those loans will default more often, and so the algorithm could "correctly" conclude that minorities default more often. Cause and effect are very hard for an algorithm to distinguish when it's the clustered input variables that are linked together for some opaque reason. The fact is, both statements in this example would be true: "higher interest is correlated with more defaults" and "minority borrowers are correlated with defaults." The algorithm doesn't know one from the other. Sure, more training data could help the algorithm distinguish between the two, but there may not be enough training data that shows the inverse -- low-interest loans given to minorities -- if the historical data is biased enough.

yummyfajitas 10y ago
Suppose the training data is a bank that has historically given higher-interest loans to minorities. Those loans will default more often, and so the algorithm could "correctly" conclude that minorities default more often. But if the relationship is actually `defaultProb = a x interest_rate + b`, then the algorithm will reflect that. I discuss this case explicitly in the blog post I linked to in the section "What if black people don't perform as well?", and explicitly give an example of a linear model NOT doing what you claim it will do. The fact is, both statements in this example would be true: "higher interest is correlated with more defaults" and "minority borrowers are correlated with defaults." The algorithm doesn't know one from the other. No, it wouldn't because you'd also have cases of white people with high interest who defaulted. The model using race as a predictive variable would fail to fit those data points. I explicitly discuss this case in my blog post. You are correct that in some cases you don't have enough training data. But in these cases, there is no particular reason for bias to have a particular sign. Again, as I discuss in the blog post I linked, why would Captain Kirk be biased against "black on left guy" vs "black on right guy"? Insufficient training data causes variance, not bias, and could just as easily result in bias in favor of blacks. (In fact, I link to a number of examples where it does.)
- 10y ago

sharemywin 10y ago

red-line - "The term refers to the presumed practice of mortgage lenders of drawing red lines around portions of a map to indicate areas or neighborhoods in which they do not want to make loans. Redlining on a racial basis has been held by the courts to be an illegal practice." What's interesting is if at some point in future regulators extend "red line" to mean variable highly correlated to a protected class. Because if your using a variable and not checking how it correlates to a protected class and making financial based decisions on it. Couldn't you in affect transform the decision into a graph "women" on on side "men" on another(or what ever your favorite protected class is).

thaw13579 10y ago

I appreciate this analysis, but it's important to note that not everyone wants unbiased model output. Historically, redlining was intentional, and it took laws to stop it. A major risk of automating mortgage engines is that redlining could easily be reinstated (in the form of model parameters) in a way that is both hard to detect and easily deniable.

yummyfajitas 10y ago
There are two possible reasons for redlining: 1) I hate black people more than I like money, but blacks do pay back their loans, so I'll redline even if it costs me money. 2) I like money and am neutral towards blacks, but blacks are deadbeats so redlining gets me more money. (There are also 2 other cases. "I love/am neutral towards blacks and they pay back their loans", which results in no disparate impact. Also "I hate blacks and they don't pay back their loans" which is equivalent to (2).) Statistical algorithms solve (1), yet somehow disparate impact isn't fixed. (2) is unspeakable to the modern left - only racists could say that - yet statistics continually reproduces conclusion (2). So obviously the problem must be with the algorithms.
- dragonwriter 10y ago
  There is also "I am neutral to blacks and view them as, ceteris paribus, equally likely to pay back loans, but white people on average are not neutral toward them, and particularly are not neutral to living in close proximity to them, and the decrease in property values when black people are allowed in otherwise white neighborhoods makes both their default and the default of white people in the neighborhood more likely, and makes deficiency more likely in the event of default of either." (Statistical models that properly account track all-factors likelihood of default on a particular loan do not solve this.)
  - yummyfajitas 10y ago
    True, a model which predicts reality will in fact predict reality. I've never seen any evidence that your hypothesis is true, or even a model that would account for it, but I'm hardly an expert in the field. Assuming this is more than speculation/theoretical nitpicking, do you have citations on the effects you describe? If your theory is more than just idle speculation, this is a great way for non-racists to engage in house price arbitrage insofar as they wish to consume housing. Do you know of any neighborhoods in NYC where I can exploit this arbitrage?