> So you're basically saying the algorithm is broken because it discovers illegal correlations, even though they may be true. No. Discovering those correlatio

> So you're basically saying the algorithm is broken because it discovers illegal correlations, even though they may be true. No. Discovering those correlations is completely legal. Making certain decisions based upon those correlations is illegal. Mostly because by making a decision, you make a tacit assumption about causation that's borderline impossible to prove and can have a huge impact on people's lives. Like I said, if it's just you in your home office having fun, go at it. But if you then bake that model into certain products, you have a very serious bug. > Well, that's precisely "polite fiction". Except in this case, it's not even that!!! The observation that certain crime statistics are highly correlated with race, and that race is correlated with zip code, are not new observations. ML did not usher in some brave new world here. Just because we call it "AI in 2018" instead of "John from the Actuarial dept. in 1960" doesn't change the moral, ethical, or legal landscape. (the article literally makes exactly this point.) And despite your characterization, this particular truth (race correlates to crime correlates to zip) isn't even a politically incorrect observation!!! It's something everyone already knows, and I've never seen someone attacked for pointing out this correlation. In fact, it's a favorite talking point of social justice types! Pointing out this correlation is not impolite. The impolite assertion is that there's a causative link between race and crime. That's an assertion that models (tacitly) make when their users shift from truth-seeking to decision-making. See the "question you thought you asked vs. question you actually asked" portion of the essay. Now, if ML algorithms discovered some genetic, racial, causal theory of crime, then you might have a point about ML exposing polite fictions in this case. But they didn't, so you don't. COMPAS isn't being censored from sharing a politically incorrect truth. It's being prevented from ruining people's lives with a really, really lazy application of statistics. I often joke that racial discrimination laws are one of the few examples where "being bad at math" is not just criminal, but unconstitutional. > ...laws are not designed as truth-seeking tools Again, in cases where bias becomes illegal, these models are NOT just being used to seek truth. They're being used to make decisions. You seem to have misconstrued the fundamental thesis of the article. The author isn't calling for death to polite fictions. And there's a concrete example in the article of this point of departure between your perspective and his. Namely, his response to people-as-gorillas was not "fuck you, the math is right". And the crescendo of the piece is "AI is just a tool, not a divine oracle, and there's nothing new under the sun".