3 ms·
I's complicated, and she's assuming a lot of high-ish level understanding of law, ML, and society, but the short answer is "Correlation doesn't imply causation,
by shadowgovt 6y ago
I's complicated, and she's assuming a lot of high-ish level understanding of law, ML, and society, but the short answer is "Correlation doesn't imply causation, but correlation is what ML systems do."
So, it may, in fact, be true that if you aggregate over statistics in a society, you'll find correlation between skin color and crime recidivism, or likelihood to embezzle money, or some other factors. It's not "okay" to bake that signal into your system for risk assessment because it creates positive feedback loops on societal inequality that penalize people for qualities out of their control, and (at least in the US) we've broadly decided that's not cool (and, in fact, narrowly codified that into law---denying people a loan based on skin color is a crime, and denying people a loan if your ML system decides "Janelle" is probably not going to make good on repayment because the name correlates to dark skin and dark skin correlates to bad loan payment history is still a crime; you can't dodge the intent of the law by adding length to the lever).
80% of the time, I don't think I'd assume someone is being racist if they act like they don't know this, but the subject of the Medium post should, ostensibly, know this.