4 ms·

> The goal is to build a model that accurately reflects the real world. This is incredibly reductive - we use models for MANY purposes - prediction, inference,

by gbrown 7y ago

> The goal is to build a model that accurately reflects the real world. This is incredibly reductive - we use models for MANY purposes - prediction, inference, description, evaluation. Until you outline a specific use case for the model, understand where the data its trained on comes from and how it may or may not reflect reality, and think about whether the phenomena which you're measuring are important/sufficient for your application you CANNOT possibly justify your broad claims.

manfredo 7y ago
I'm well aware that models are used for a variety of applications. But regardless of the application, we usually want the model to reflect reality in some way. Say you have an AI that predicts probability of re-offending for prospective parolees. If we observe that this system is flagging men as high risk more frequently than women, then this is not indicative of bias. But!, one might say, men are socialized to be more violent and are more frequently recruited by gangs. These factors are being absorbed by the AI. Yes, they are, and this does not make the AI biased. The job of the AI to predict risk of re-offending in real world conditions. Not risk of re-offending in a hypothetical world where everyone has the same environment and experiences. If we were to manually adjust the model to make it display equal risks for both men and women then that is not elimination of bias. It is deliberate introduction of bias to make the output fit our ideal worldview.
- crooked-v 7y ago
  > If we observe that this system is flagging men as high risk more frequently than women, then this is not indicative of bias. If all it's doing is associating being a man with a higher risk factor without taking into account anything else in that man's life then, yes, it's being biased. "Being a man" in this situation is a correlation to, not casuation of, actual factors making a person more likely to commit crime. This is a real problem that has been happening for years in Florida: the automated risk assessments used for a variety of decisions, including bail and sentencing factors, labels black people as inherently being more likely to reoffend than white people, even after controlling for actual crime rates and recidivism rates. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm...
  - manfredo 7y ago
    It depends on the crime. But for crimes like, say, sexual assault being male is indeed a huge factor in likelihood to offend. Disparities between rates of women committing sexual assault and rates of men is immense - one or two orders of magnitude depending on the exact crimes counted. I agree that an AI shouldn't use race or gender as an input. But we should not surprised when AIs have disparities when predicting risk of committing crimes, when there are disparities in the rates of actually committing crimes.
  - losvedir 7y ago
    That propublica article is the perfect example of the many aspects of the discussion here. It argues just one side of it, but the scenario it raises is one where an ML algorithm correctly found a solution to a badly framed problem. Briefly, (and simplifying for clarity) it worked like this: the ML algorithm scored the risk of criminals reoffending. It gave a "high risk" score if someone was 80% likely to re-offend, and a "low risk" score if they had a 20% chance. Out of 100 blacks and 100 whites, here are the scores the ML algorithm gave, and how many ultimately did re-offend: 100 black criminals * 50 "high risk", 40 re-offended * 50 "low risk", 10 re-offended 100 white criminals * 10 "high risk", 8 re-offended * 90 "low risk", 18 re-offended On the one hand, the algorithm was completely correct: 80% of high risk individuals re-offended, and 20% of low risk individuals did, and this is true even looking at just the black or the white criminals. It was unbiased according to its goals. On the other hand, the failure mode disproportionately punished the black criminals: 10 were "high risk" who never re-offended, while only 2 white criminals were. Meanwhile, 18 "low risk" white criminals did re-offend while only 10 black criminals did. So the score was more strict for some unlucky blacks and more lenient for some lucky whites. However, a key point is that because the underlying re-offense rates were different between the populations (50/100 for the black criminals, 26/100 for the white criminals), the algorithm could not have done otherwise. That is, given some n% re-offenders, if you have to fit them into 20% and 80% buckets, your "high risk" and "low risk" counts are mathematically fixed. In other words, it wasn't the ML by itself that was the problem, but the COMPAS score that it was trying to compute that had these issues inherent in it. I think this is a good example where ML wasn't biased (at least, not anymore than reality), but where people were too eager to turn to it for a poorly considered project. By wrapping up important questions in high tech algorithms, it's too easy to fool yourself that what you're doing is the best thing you could be doing and miss problems in the fundamental framing.
- gbrown 7y ago
  > Say you have an AI that predicts probability of re-offending for prospective parolees. Already you've made one of the most common mistakes in this area: confusing the label for the latent truth. We don't measure "re-offending", we measure rearrest and re-incarceration, which as I've discussed elsewhere are biased (in the sense of "unjust"). > If we were to manually adjust the model to make it display equal risks for both men and women then that is not elimination of bias. It is deliberate introduction of bias to make the output fit our ideal worldview. Again, you are using the word "bias" in a vague and meaningless way. In this context, our goals are constrained by the law - in the US we guarantee people certain rights, and prohibit certain kinds of discrimination. In this example, it would indeed be wrong (and illegal) to punish men more harshly just for being men, since gender is a protected class. In the same way, it would be wrong (and illegal) to punish black people more harshly just for being black. In my state, whites and blacks use cannabis at equivalent rates, and black people are arrested 8 times as often per capita - an algorithm looking at arrests will see that and make the problem worse. You've staked out some sort of naiive and bizarre "AI Purity" stance which completely ignores how such models work, and which misunderstands how we use models to learn things and solve problems. You're also mixing up different definitions of "bias". Some facts which you might find interesting: * Penalization/shrinkage, which is ubiquitous and incredibly useful in many predictive models, is the deliberate introduction of bias to improve prediction performance * Adjusting for confounding in regression/classification type models is one way of accounting for or removing bias in a statistically rigorous way - as you (imprecisely and inaccurately) say, "modifying the model to make it display equal risks". This allows us to measure effects, conditional on known risks or existing patterns. * All statistical and machine learning algorithms suffer from sampling bias, where the data we observe doesn't match the reality in important ways - a fact which you're completely ignoring. If a model reflects the bias in a bad sample, why on earth should we accept that?

by gbrown 7y ago

manfredo 7y ago

I'm well aware that models are used for a variety of applications. But regardless of the application, we usually want the model to reflect reality in some way. Say you have an AI that predicts probability of re-offending for prospective parolees. If we observe that this system is flagging men as high risk more frequently than women, then this is not indicative of bias. But!, one might say, men are socialized to be more violent and are more frequently recruited by gangs. These factors are being absorbed by the AI. Yes, they are, and this does not make the AI biased. The job of the AI to predict risk of re-offending in real world conditions. Not risk of re-offending in a hypothetical world where everyone has the same environment and experiences. If we were to manually adjust the model to make it display equal risks for both men and women then that is not elimination of bias. It is deliberate introduction of bias to make the output fit our ideal worldview.

crooked-v 7y ago
> If we observe that this system is flagging men as high risk more frequently than women, then this is not indicative of bias. If all it's doing is associating being a man with a higher risk factor without taking into account anything else in that man's life then, yes, it's being biased. "Being a man" in this situation is a correlation to, not casuation of, actual factors making a person more likely to commit crime. This is a real problem that has been happening for years in Florida: the automated risk assessments used for a variety of decisions, including bail and sentencing factors, labels black people as inherently being more likely to reoffend than white people, even after controlling for actual crime rates and recidivism rates. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm...
- manfredo 7y ago
  It depends on the crime. But for crimes like, say, sexual assault being male is indeed a huge factor in likelihood to offend. Disparities between rates of women committing sexual assault and rates of men is immense - one or two orders of magnitude depending on the exact crimes counted. I agree that an AI shouldn't use race or gender as an input. But we should not surprised when AIs have disparities when predicting risk of committing crimes, when there are disparities in the rates of actually committing crimes.
- losvedir 7y ago
  That propublica article is the perfect example of the many aspects of the discussion here. It argues just one side of it, but the scenario it raises is one where an ML algorithm correctly found a solution to a badly framed problem. Briefly, (and simplifying for clarity) it worked like this: the ML algorithm scored the risk of criminals reoffending. It gave a "high risk" score if someone was 80% likely to re-offend, and a "low risk" score if they had a 20% chance. Out of 100 blacks and 100 whites, here are the scores the ML algorithm gave, and how many ultimately did re-offend: 100 black criminals * 50 "high risk", 40 re-offended * 50 "low risk", 10 re-offended 100 white criminals * 10 "high risk", 8 re-offended * 90 "low risk", 18 re-offended On the one hand, the algorithm was completely correct: 80% of high risk individuals re-offended, and 20% of low risk individuals did, and this is true even looking at just the black or the white criminals. It was unbiased according to its goals. On the other hand, the failure mode disproportionately punished the black criminals: 10 were "high risk" who never re-offended, while only 2 white criminals were. Meanwhile, 18 "low risk" white criminals did re-offend while only 10 black criminals did. So the score was more strict for some unlucky blacks and more lenient for some lucky whites. However, a key point is that because the underlying re-offense rates were different between the populations (50/100 for the black criminals, 26/100 for the white criminals), the algorithm could not have done otherwise. That is, given some n% re-offenders, if you have to fit them into 20% and 80% buckets, your "high risk" and "low risk" counts are mathematically fixed. In other words, it wasn't the ML by itself that was the problem, but the COMPAS score that it was trying to compute that had these issues inherent in it. I think this is a good example where ML wasn't biased (at least, not anymore than reality), but where people were too eager to turn to it for a poorly considered project. By wrapping up important questions in high tech algorithms, it's too easy to fool yourself that what you're doing is the best thing you could be doing and miss problems in the fundamental framing.
gbrown 7y ago
> Say you have an AI that predicts probability of re-offending for prospective parolees. Already you've made one of the most common mistakes in this area: confusing the label for the latent truth. We don't measure "re-offending", we measure rearrest and re-incarceration, which as I've discussed elsewhere are biased (in the sense of "unjust"). > If we were to manually adjust the model to make it display equal risks for both men and women then that is not elimination of bias. It is deliberate introduction of bias to make the output fit our ideal worldview. Again, you are using the word "bias" in a vague and meaningless way. In this context, our goals are constrained by the law - in the US we guarantee people certain rights, and prohibit certain kinds of discrimination. In this example, it would indeed be wrong (and illegal) to punish men more harshly just for being men, since gender is a protected class. In the same way, it would be wrong (and illegal) to punish black people more harshly just for being black. In my state, whites and blacks use cannabis at equivalent rates, and black people are arrested 8 times as often per capita - an algorithm looking at arrests will see that and make the problem worse. You've staked out some sort of naiive and bizarre "AI Purity" stance which completely ignores how such models work, and which misunderstands how we use models to learn things and solve problems. You're also mixing up different definitions of "bias". Some facts which you might find interesting: * Penalization/shrinkage, which is ubiquitous and incredibly useful in many predictive models, is the deliberate introduction of bias to improve prediction performance * Adjusting for confounding in regression/classification type models is one way of accounting for or removing bias in a statistically rigorous way - as you (imprecisely and inaccurately) say, "modifying the model to make it display equal risks". This allows us to measure effects, conditional on known risks or existing patterns. * All statistical and machine learning algorithms suffer from sampling bias, where the data we observe doesn't match the reality in important ways - a fact which you're completely ignoring. If a model reflects the bias in a bad sample, why on earth should we accept that?