4 ms·

But that loan officer brings their own bias to the scenario. They could just as easily say the Johnson's are unreliable because they are black. It wasn't that l

But that loan officer brings their own bias to the scenario. They could just as easily say the Johnson's are unreliable because they are black. It wasn't that long ago that saying that was institutionalized and I still suspect it occurs. An algorithm is colorblind. This isn't to say an algorithm is necessarily good, but that humans aren't either. One of the reasons bureaucratic red tape exists is as an effort to overcome individual judgement in favor of consistent and fair judgement. So though there's human intervention or policy considerations that need to occur in this situation due to massive abuses and obvious unfairness, I don't think it damns the mathematical model as a concept.

paulmd 10y ago
That's really what I'm arguing overall, mathematical models are great as a second opinion, or even a first opinion, but at some point you do really need a human in the loop to say "nah that doesn't make sense". The nice thing about this, versus - say - a self-driving car - is that you don't have to make a decision within 0.4 seconds before the car crashes. There is plenty of time to get a human into the loop here, and to show the human why the model thinks what it does. Models that can't explain themselves, well, that's going to be of limited social value unless the need is absolutely dire like the car scenario. If the car gets me out of a crash, great, but I would want to know why my loan was denied. If you can't put it into words or show a plot of the classifier... well... condorcet voting hasn't taken off either, everyone seems to prefer IRV so far even if it's "less optimal". We have plenty of experience with "if you screw up as loan officer too much you lose your job". That's a pretty well-proven model. Especially now that you have an additional guidance signal on when the person is going "off the reservation" so to speak. Be a loose cannon too much and you're going to be the first guy carrying their stuff out the door in a box. Frankly I'm much more enthused about the reverse here - I want to see the model tell me who's a bad loan officer given the hand they're dealt, not what a bad loan is. We can even incorporate social outcomes into that scoring metric. It's a lot like teachers. Would you try to black-box yourself to the perfect textbook? Or would you be better off trying to figure out who's the struggling teacher in a shitty school and who's the lazy teacher half-assing it in the rich 'burbs?

paulmd 10y ago

That's really what I'm arguing overall, mathematical models are great as a second opinion, or even a first opinion, but at some point you do really need a human in the loop to say "nah that doesn't make sense". The nice thing about this, versus - say - a self-driving car - is that you don't have to make a decision within 0.4 seconds before the car crashes. There is plenty of time to get a human into the loop here, and to show the human why the model thinks what it does. Models that can't explain themselves, well, that's going to be of limited social value unless the need is absolutely dire like the car scenario. If the car gets me out of a crash, great, but I would want to know why my loan was denied. If you can't put it into words or show a plot of the classifier... well... condorcet voting hasn't taken off either, everyone seems to prefer IRV so far even if it's "less optimal". We have plenty of experience with "if you screw up as loan officer too much you lose your job". That's a pretty well-proven model. Especially now that you have an additional guidance signal on when the person is going "off the reservation" so to speak. Be a loose cannon too much and you're going to be the first guy carrying their stuff out the door in a box. Frankly I'm much more enthused about the reverse here - I want to see the model tell me who's a bad loan officer given the hand they're dealt, not what a bad loan is. We can even incorporate social outcomes into that scoring metric. It's a lot like teachers. Would you try to black-box yourself to the perfect textbook? Or would you be better off trying to figure out who's the struggling teacher in a shitty school and who's the lazy teacher half-assing it in the rich 'burbs?