4 ms·

Ah, yes, but that's not really the point, is it? COMPAS points out an uncomfortable trend ("violent crime is more prevalent in black communities...") but appli

Ah, yes, but that's not really the point, is it? COMPAS points out an uncomfortable trend ("violent crime is more prevalent in black communities...") but applies it in a way that is bad ("...and therefore black people should get longer criminal sentences for the same crime"). We both agree, I'm sure, that socioeconomics and race shouldn't meaningfully impact sentence length. We may also agree that it often does. What a lot of these models do is encode that outcome (that bias, some might say) in seemingly 'objective' systems. That reproduction of bias is why they're discriminatory - not because they look at crime stats, but because they apply them in a way that most people wouldn't.

elisbce 6y ago
You are deliberately twisting it. The crime stats says "violent crime is more prevalent in black communities...", but no AI/ML will produce the conclusion "...and therefore black people should get longer criminal sentences for the same crime". They are completely irrelevant. The AI model will simply conclude that race would be a factor when you are looking at crime rate, or posterior probability when a violent crime happens. And that is exactly accurate. What's wrong with that? That's a fact. Math doesn't lie nor does it have any agenda. There is nothing the AI is making up or adding up to. It is pure statistics and probability. It is the human beings that can't take the truth and the consequences.
- curiousllama 6y ago
  You're right, of course: "human beings cant take the truth and consequences." "That is exactly accurate [...] That's a fact" - both true. I think the issue is less truth and more action, as in "I believe it to be immoral to act upon that fact in manner X." I can come up with a dozen logical statements that are equally correct and immoral to act upon in certain ways. If an ML algorithm had come up with them, they would be no less correct and no less immoral. For example, "If a woman applicant is pregnant, she will take maternity leave and be less productive year 1. Therefore, we should not hire her." <- likely correct, definitely immoral (imo). COMPAS drew a number of objective conclusions and ties them to specific recommendations in its intended use: (1) if you're poor, you're more likely to recidivate. Therefore, you should get a longer sentence. (2) if you live in a black community, you're more likely to recidivate. Therefore, you should get a longer sentence (3) if you're related to a gang member (not just one yourself - even if your estranged brother joined a gang a decade ago), you're more likely to recidivate. Therefore, you should get a longer sentence. Notice: the "you should get a longer sentence" bit is actually part of the tool. That was the intervention it was designed to suggest. There's a number of critiques of that connection (mathematical, legal, moral), but it's not mine, nor is it the "AI's"; it's Northpointe's. And ultimately that's the scandal & intrigue: the math is right logically, but even so, the tool's recommendation is wrong ethically.
- Hnrobert42 6y ago
  The problem is that enforcement begets enforcement. Once one is entangled in the legal system, they are more likely to be called in for questioning, have their fingerprints possibly match a partial set left at a crime, have a possible DNA match turn into a warrant that leads to arrest on an unrelated crime. This is true because the web of laws, in the US at least, means that everyone breaks dozens of laws a day. So AI becomes a self-licking ice cream cone. Even worse, it is one that comes with a plausible argument of simply being an unbiased mathematical conclusion. Of course math doesn’t have an agenda, but we aren’t talking about the actual algorithm having a human bias. We are talking about the application of math by humans to human problems using data sets curated by humans. Using math doesn’t purge the biases of all those humans from the system.

elisbce 6y ago

You are deliberately twisting it. The crime stats says "violent crime is more prevalent in black communities...", but no AI/ML will produce the conclusion "...and therefore black people should get longer criminal sentences for the same crime". They are completely irrelevant. The AI model will simply conclude that race would be a factor when you are looking at crime rate, or posterior probability when a violent crime happens. And that is exactly accurate. What's wrong with that? That's a fact. Math doesn't lie nor does it have any agenda. There is nothing the AI is making up or adding up to. It is pure statistics and probability. It is the human beings that can't take the truth and the consequences.

curiousllama 6y ago
You're right, of course: "human beings cant take the truth and consequences." "That is exactly accurate [...] That's a fact" - both true. I think the issue is less truth and more action, as in "I believe it to be immoral to act upon that fact in manner X." I can come up with a dozen logical statements that are equally correct and immoral to act upon in certain ways. If an ML algorithm had come up with them, they would be no less correct and no less immoral. For example, "If a woman applicant is pregnant, she will take maternity leave and be less productive year 1. Therefore, we should not hire her." <- likely correct, definitely immoral (imo). COMPAS drew a number of objective conclusions and ties them to specific recommendations in its intended use: (1) if you're poor, you're more likely to recidivate. Therefore, you should get a longer sentence. (2) if you live in a black community, you're more likely to recidivate. Therefore, you should get a longer sentence (3) if you're related to a gang member (not just one yourself - even if your estranged brother joined a gang a decade ago), you're more likely to recidivate. Therefore, you should get a longer sentence. Notice: the "you should get a longer sentence" bit is actually part of the tool. That was the intervention it was designed to suggest. There's a number of critiques of that connection (mathematical, legal, moral), but it's not mine, nor is it the "AI's"; it's Northpointe's. And ultimately that's the scandal & intrigue: the math is right logically, but even so, the tool's recommendation is wrong ethically.
Hnrobert42 6y ago
The problem is that enforcement begets enforcement. Once one is entangled in the legal system, they are more likely to be called in for questioning, have their fingerprints possibly match a partial set left at a crime, have a possible DNA match turn into a warrant that leads to arrest on an unrelated crime. This is true because the web of laws, in the US at least, means that everyone breaks dozens of laws a day. So AI becomes a self-licking ice cream cone. Even worse, it is one that comes with a plausible argument of simply being an unbiased mathematical conclusion. Of course math doesn’t have an agenda, but we aren’t talking about the actual algorithm having a human bias. We are talking about the application of math by humans to human problems using data sets curated by humans. Using math doesn’t purge the biases of all those humans from the system.