3 ms·

Ehh.. the discrimination part sounds a lot like: "we had an AI look at statistics and it reported the patterns we choose to ignore".

jl6 6y ago
AIs produce correlation which gets sold as causation.
- 6gvONxR4sf7o 6y ago
  Which is then justified as causation we chose to ignore for political correctness.
- centimeter 6y ago
  A lot of the patterns these things pick up on are causal and we just don't like the fact that there are real causal networks that conflict with our religious/idealogical beliefs.
curiousllama 6y ago
Ah, yes, but that's not really the point, is it? COMPAS points out an uncomfortable trend ("violent crime is more prevalent in black communities...") but applies it in a way that is bad ("...and therefore black people should get longer criminal sentences for the same crime"). We both agree, I'm sure, that socioeconomics and race shouldn't meaningfully impact sentence length. We may also agree that it often does. What a lot of these models do is encode that outcome (that bias, some might say) in seemingly 'objective' systems. That reproduction of bias is why they're discriminatory - not because they look at crime stats, but because they apply them in a way that most people wouldn't.
- elisbce 6y ago
  You are deliberately twisting it. The crime stats says "violent crime is more prevalent in black communities...", but no AI/ML will produce the conclusion "...and therefore black people should get longer criminal sentences for the same crime". They are completely irrelevant. The AI model will simply conclude that race would be a factor when you are looking at crime rate, or posterior probability when a violent crime happens. And that is exactly accurate. What's wrong with that? That's a fact. Math doesn't lie nor does it have any agenda. There is nothing the AI is making up or adding up to. It is pure statistics and probability. It is the human beings that can't take the truth and the consequences.
  - curiousllama 6y ago
    You're right, of course: "human beings cant take the truth and consequences." "That is exactly accurate [...] That's a fact" - both true. I think the issue is less truth and more action, as in "I believe it to be immoral to act upon that fact in manner X." I can come up with a dozen logical statements that are equally correct and immoral to act upon in certain ways. If an ML algorithm had come up with them, they would be no less correct and no less immoral. For example, "If a woman applicant is pregnant, she will take maternity leave and be less productive year 1. Therefore, we should not hire her." <- likely correct, definitely immoral (imo). COMPAS drew a number of objective conclusions and ties them to specific recommendations in its intended use: (1) if you're poor, you're more likely to recidivate. Therefore, you should get a longer sentence. (2) if you live in a black community, you're more likely to recidivate. Therefore, you should get a longer sentence (3) if you're related to a gang member (not just one yourself - even if your estranged brother joined a gang a decade ago), you're more likely to recidivate. Therefore, you should get a longer sentence. Notice: the "you should get a longer sentence" bit is actually part of the tool. That was the intervention it was designed to suggest. There's a number of critiques of that connection (mathematical, legal, moral), but it's not mine, nor is it the "AI's"; it's Northpointe's. And ultimately that's the scandal & intrigue: the math is right logically, but even so, the tool's recommendation is wrong ethically.
  - Hnrobert42 6y ago
    The problem is that enforcement begets enforcement. Once one is entangled in the legal system, they are more likely to be called in for questioning, have their fingerprints possibly match a partial set left at a crime, have a possible DNA match turn into a warrant that leads to arrest on an unrelated crime. This is true because the web of laws, in the US at least, means that everyone breaks dozens of laws a day. So AI becomes a self-licking ice cream cone. Even worse, it is one that comes with a plausible argument of simply being an unbiased mathematical conclusion. Of course math doesn’t have an agenda, but we aren’t talking about the actual algorithm having a human bias. We are talking about the application of math by humans to human problems using data sets curated by humans. Using math doesn’t purge the biases of all those humans from the system.
lxe 6y ago
The issue is using the correlation to make predictions. You're just perpetuating the trend instead of trying to change it.
- elisbce 6y ago
  It has nothing to do with AI/ML. If you see someone repeatedly stealing from your yard, you can't predict he would do it again with high probability, because it would perpetuate the trend instead of trying to change it? Why would the responsibility of change lie on the observers instead of the perpetrators? If he cares about his image, he can stop stealing and with time, people would conclude so.

jl6 6y ago

AIs produce correlation which gets sold as causation.

6gvONxR4sf7o 6y ago
Which is then justified as causation we chose to ignore for political correctness.
centimeter 6y ago
A lot of the patterns these things pick up on are causal and we just don't like the fact that there are real causal networks that conflict with our religious/idealogical beliefs.

curiousllama 6y ago

Ah, yes, but that's not really the point, is it? COMPAS points out an uncomfortable trend ("violent crime is more prevalent in black communities...") but applies it in a way that is bad ("...and therefore black people should get longer criminal sentences for the same crime"). We both agree, I'm sure, that socioeconomics and race shouldn't meaningfully impact sentence length. We may also agree that it often does. What a lot of these models do is encode that outcome (that bias, some might say) in seemingly 'objective' systems. That reproduction of bias is why they're discriminatory - not because they look at crime stats, but because they apply them in a way that most people wouldn't.

elisbce 6y ago
You are deliberately twisting it. The crime stats says "violent crime is more prevalent in black communities...", but no AI/ML will produce the conclusion "...and therefore black people should get longer criminal sentences for the same crime". They are completely irrelevant. The AI model will simply conclude that race would be a factor when you are looking at crime rate, or posterior probability when a violent crime happens. And that is exactly accurate. What's wrong with that? That's a fact. Math doesn't lie nor does it have any agenda. There is nothing the AI is making up or adding up to. It is pure statistics and probability. It is the human beings that can't take the truth and the consequences.
- curiousllama 6y ago
  You're right, of course: "human beings cant take the truth and consequences." "That is exactly accurate [...] That's a fact" - both true. I think the issue is less truth and more action, as in "I believe it to be immoral to act upon that fact in manner X." I can come up with a dozen logical statements that are equally correct and immoral to act upon in certain ways. If an ML algorithm had come up with them, they would be no less correct and no less immoral. For example, "If a woman applicant is pregnant, she will take maternity leave and be less productive year 1. Therefore, we should not hire her." <- likely correct, definitely immoral (imo). COMPAS drew a number of objective conclusions and ties them to specific recommendations in its intended use: (1) if you're poor, you're more likely to recidivate. Therefore, you should get a longer sentence. (2) if you live in a black community, you're more likely to recidivate. Therefore, you should get a longer sentence (3) if you're related to a gang member (not just one yourself - even if your estranged brother joined a gang a decade ago), you're more likely to recidivate. Therefore, you should get a longer sentence. Notice: the "you should get a longer sentence" bit is actually part of the tool. That was the intervention it was designed to suggest. There's a number of critiques of that connection (mathematical, legal, moral), but it's not mine, nor is it the "AI's"; it's Northpointe's. And ultimately that's the scandal & intrigue: the math is right logically, but even so, the tool's recommendation is wrong ethically.
- Hnrobert42 6y ago
  The problem is that enforcement begets enforcement. Once one is entangled in the legal system, they are more likely to be called in for questioning, have their fingerprints possibly match a partial set left at a crime, have a possible DNA match turn into a warrant that leads to arrest on an unrelated crime. This is true because the web of laws, in the US at least, means that everyone breaks dozens of laws a day. So AI becomes a self-licking ice cream cone. Even worse, it is one that comes with a plausible argument of simply being an unbiased mathematical conclusion. Of course math doesn’t have an agenda, but we aren’t talking about the actual algorithm having a human bias. We are talking about the application of math by humans to human problems using data sets curated by humans. Using math doesn’t purge the biases of all those humans from the system.

lxe 6y ago

The issue is using the correlation to make predictions. You're just perpetuating the trend instead of trying to change it.

elisbce 6y ago
It has nothing to do with AI/ML. If you see someone repeatedly stealing from your yard, you can't predict he would do it again with high probability, because it would perpetuate the trend instead of trying to change it? Why would the responsibility of change lie on the observers instead of the perpetrators? If he cares about his image, he can stop stealing and with time, people would conclude so.