4 ms·

So when an ML algorithm spits out something that goes against the prevailing ideologies, it's most likely the fact in the data. You seem to have missed the sub

So when an ML algorithm spits out something that goes against the prevailing ideologies, it's most likely the fact in the data. You seem to have missed the subtlety in the author's treatment of this subject. His point is exactly that the data is not necessarily "good". It's biased because we are biased [1], and you have to be aware of that when interpreting results. It's the data that needs to be questioned. [1] if three white teenagers are arrested for a crime, not only are news media much less likely to show their mug shots, but they’re less likely to refer to them as “white teenagers.”

TeMPOraL 9y ago
I didn't miss it; I address it in the very next sentence - "If the data is good (i.e. it represents the real world properly), then the ideologies should be questioned." There is a if-then clause there. The point about mugshots shows perfectly the problem of what happens when the data is biased. But then the corollary is, sometimes the data is good. The relevant quote from article: > If you try to manually “ignore race” by not letting race be an input to your model, it comes in through the back door: for example, someone’s zip code and income predict their race with great precision. This is what happens with loan or crime data sets, and it made plenty of noise in the news recently - but only the kind of noise in which people say it's obviously the algorithm that's broken, because it doesn't fit the "polite fiction" they'd like to believe.
- danso 9y ago
  Huh? It seems to me that the OP is using the example of race correlating with zip code as another example of how data can be misused and misinterpreted, e.g. feeding a dataset that has zip code and income could still lead to an AI making a model not much different than if the data had race in it.
  - TeMPOraL 9y ago
    I believe the article implies that calling it "misuse" or "misinterpretation" is preferring "polite fiction" to reality.
    - danso 9y ago
      The text does not support your interpretation. He says that data in the United States almost always has a racial bias, and that even when you explicitly remove race from the data, you and your ML model can still inadvertently make predictions that fall along racial lines because of how other data points strongly correlate to race. He then goes on to talk about COMPAS, which was accused of being racially biased even though COMPAS explicitly did not include race as one of its inputs. The author argues that COMPAS still mad predictions that were racially biased because of how strong the correlation is between race and housing. On top of that, the author argues, COMPAS asked the wrong question (who is likely to be more convicted, vs who is more likely to commit crime). The author does not, as you seem to do, celebrate the Justice system algorithms for making harsher, but objective data-driven recommendations for minority groups. He argues that this is unjust, that the AI is missing the necessary nuance and context, and that the courts should not have been so trusting of AI: > There are obviously many problems at play here. One is that the courts took the AI model far too seriously, using it as a direct factor in sentencing decisions, skipping human judgment, with far more confidence than any model should warrant. (A good rule of thumb, also recently encoded into EU law, is that decisions with serious consequences of people should be sanity-checked by a human — and that there should be a human override mechanism available.) Another problem, of course, is the underlying systemic racism which this exposed: the fact that Black people are more likely to be arrested and convicted of the same crimes.
- throwawayjava 9y ago
  > But then the corollary is, sometimes the data is good. Yes. but in my limited experience with ML, "the data is good" usually isn't the >> most likely explanation :-) > This is what happens with loan or crime data sets, and it made plenty of noise in the news recently - but only the kind of noise in which people say it's obviously the algorithm that's broken, because it doesn't fit the "polite fiction" they'd like to believe. People say the algorithm is broken because it's illegal to discriminate on the basis of race, and these algorithms were sneaking "discriminate by race" "in through the back door". Calling the law a polite fiction won't get you very far when the judge issues an injunction against using your product because it's racially biased. And the judge doesn't care how machine learning researchers define bias. He cares how the law defines bias. So, if you're diddling around on your computer in your own time, then I guess the algorithm isn't broken. But if you're building a product you want to sell to courts or insurance companies, the algorithm very much is broken. That is, unless you think "the law is a polite fiction and bias means only what ML researchers say it means" would be a winning argument to lift an injunction against your product. If you're going to make those arguments in front of a real judge, let me know; I want to see the bulge in that judge's forehead :-) No. People say the algorithm is broken because it obviously is broken.
  - TeMPOraL 9y ago
    So you're basically saying the algorithm is broken because it discovers illegal correlations, even though they may be true. Well, that's precisely "polite fiction". (Also, laws are not designed as truth-seeking tools but usable heuristics that take into account limits of individual humans.)
    - throwawayjava 9y ago
      > So you're basically saying the algorithm is broken because it discovers illegal correlations, even though they may be true. No. Discovering those correlations is completely legal. Making certain decisions based upon those correlations is illegal. Mostly because by making a decision, you make a tacit assumption about causation that's borderline impossible to prove and can have a huge impact on people's lives. Like I said, if it's just you in your home office having fun, go at it. But if you then bake that model into certain products, you have a very serious bug. > Well, that's precisely "polite fiction". Except in this case, it's not even that!!! The observation that certain crime statistics are highly correlated with race, and that race is correlated with zip code, are not new observations. ML did not usher in some brave new world here. Just because we call it "AI in 2018" instead of "John from the Actuarial dept. in 1960" doesn't change the moral, ethical, or legal landscape. (the article literally makes exactly this point.) And despite your characterization, this particular truth (race correlates to crime correlates to zip) isn't even a politically incorrect observation!!! It's something everyone already knows, and I've never seen someone attacked for pointing out this correlation. In fact, it's a favorite talking point of social justice types! Pointing out this correlation is not impolite. The impolite assertion is that there's a causative link between race and crime. That's an assertion that models (tacitly) make when their users shift from truth-seeking to decision-making. See the "question you thought you asked vs. question you actually asked" portion of the essay. Now, if ML algorithms discovered some genetic, racial, causal theory of crime, then you might have a point about ML exposing polite fictions in this case. But they didn't, so you don't. COMPAS isn't being censored from sharing a politically incorrect truth. It's being prevented from ruining people's lives with a really, really lazy application of statistics. I often joke that racial discrimination laws are one of the few examples where "being bad at math" is not just criminal, but unconstitutional. > ...laws are not designed as truth-seeking tools Again, in cases where bias becomes illegal, these models are NOT just being used to seek truth. They're being used to make decisions. You seem to have misconstrued the fundamental thesis of the article. The author isn't calling for death to polite fictions. And there's a concrete example in the article of this point of departure between your perspective and his. Namely, his response to people-as-gorillas was not "fuck you, the math is right". And the crescendo of the piece is "AI is just a tool, not a divine oracle, and there's nothing new under the sun".
- TheCoelacanth 9y ago
  > This is what happens with loan or crime data sets, and it made plenty of noise in the news recently - but only the kind of noise in which people say it's obviously the algorithm that's broken, because it doesn't fit the "polite fiction" they'd like to believe. The polite fiction would be that it is only the algorithm that's broken when it is actually the country that is broken.
  - TeMPOraL 9y ago
    Yes, exactly.

TeMPOraL 9y ago

I didn't miss it; I address it in the very next sentence - "If the data is good (i.e. it represents the real world properly), then the ideologies should be questioned." There is a if-then clause there. The point about mugshots shows perfectly the problem of what happens when the data is biased. But then the corollary is, sometimes the data is good. The relevant quote from article: > If you try to manually “ignore race” by not letting race be an input to your model, it comes in through the back door: for example, someone’s zip code and income predict their race with great precision. This is what happens with loan or crime data sets, and it made plenty of noise in the news recently - but only the kind of noise in which people say it's obviously the algorithm that's broken, because it doesn't fit the "polite fiction" they'd like to believe.

danso 9y ago
Huh? It seems to me that the OP is using the example of race correlating with zip code as another example of how data can be misused and misinterpreted, e.g. feeding a dataset that has zip code and income could still lead to an AI making a model not much different than if the data had race in it.
- TeMPOraL 9y ago
  I believe the article implies that calling it "misuse" or "misinterpretation" is preferring "polite fiction" to reality.
  - danso 9y ago
    The text does not support your interpretation. He says that data in the United States almost always has a racial bias, and that even when you explicitly remove race from the data, you and your ML model can still inadvertently make predictions that fall along racial lines because of how other data points strongly correlate to race. He then goes on to talk about COMPAS, which was accused of being racially biased even though COMPAS explicitly did not include race as one of its inputs. The author argues that COMPAS still mad predictions that were racially biased because of how strong the correlation is between race and housing. On top of that, the author argues, COMPAS asked the wrong question (who is likely to be more convicted, vs who is more likely to commit crime). The author does not, as you seem to do, celebrate the Justice system algorithms for making harsher, but objective data-driven recommendations for minority groups. He argues that this is unjust, that the AI is missing the necessary nuance and context, and that the courts should not have been so trusting of AI: > There are obviously many problems at play here. One is that the courts took the AI model far too seriously, using it as a direct factor in sentencing decisions, skipping human judgment, with far more confidence than any model should warrant. (A good rule of thumb, also recently encoded into EU law, is that decisions with serious consequences of people should be sanity-checked by a human — and that there should be a human override mechanism available.) Another problem, of course, is the underlying systemic racism which this exposed: the fact that Black people are more likely to be arrested and convicted of the same crimes.
throwawayjava 9y ago
> But then the corollary is, sometimes the data is good. Yes. but in my limited experience with ML, "the data is good" usually isn't the >> most likely explanation :-) > This is what happens with loan or crime data sets, and it made plenty of noise in the news recently - but only the kind of noise in which people say it's obviously the algorithm that's broken, because it doesn't fit the "polite fiction" they'd like to believe. People say the algorithm is broken because it's illegal to discriminate on the basis of race, and these algorithms were sneaking "discriminate by race" "in through the back door". Calling the law a polite fiction won't get you very far when the judge issues an injunction against using your product because it's racially biased. And the judge doesn't care how machine learning researchers define bias. He cares how the law defines bias. So, if you're diddling around on your computer in your own time, then I guess the algorithm isn't broken. But if you're building a product you want to sell to courts or insurance companies, the algorithm very much is broken. That is, unless you think "the law is a polite fiction and bias means only what ML researchers say it means" would be a winning argument to lift an injunction against your product. If you're going to make those arguments in front of a real judge, let me know; I want to see the bulge in that judge's forehead :-) No. People say the algorithm is broken because it obviously is broken.
- TeMPOraL 9y ago
  So you're basically saying the algorithm is broken because it discovers illegal correlations, even though they may be true. Well, that's precisely "polite fiction". (Also, laws are not designed as truth-seeking tools but usable heuristics that take into account limits of individual humans.)
  - throwawayjava 9y ago
    > So you're basically saying the algorithm is broken because it discovers illegal correlations, even though they may be true. No. Discovering those correlations is completely legal. Making certain decisions based upon those correlations is illegal. Mostly because by making a decision, you make a tacit assumption about causation that's borderline impossible to prove and can have a huge impact on people's lives. Like I said, if it's just you in your home office having fun, go at it. But if you then bake that model into certain products, you have a very serious bug. > Well, that's precisely "polite fiction". Except in this case, it's not even that!!! The observation that certain crime statistics are highly correlated with race, and that race is correlated with zip code, are not new observations. ML did not usher in some brave new world here. Just because we call it "AI in 2018" instead of "John from the Actuarial dept. in 1960" doesn't change the moral, ethical, or legal landscape. (the article literally makes exactly this point.) And despite your characterization, this particular truth (race correlates to crime correlates to zip) isn't even a politically incorrect observation!!! It's something everyone already knows, and I've never seen someone attacked for pointing out this correlation. In fact, it's a favorite talking point of social justice types! Pointing out this correlation is not impolite. The impolite assertion is that there's a causative link between race and crime. That's an assertion that models (tacitly) make when their users shift from truth-seeking to decision-making. See the "question you thought you asked vs. question you actually asked" portion of the essay. Now, if ML algorithms discovered some genetic, racial, causal theory of crime, then you might have a point about ML exposing polite fictions in this case. But they didn't, so you don't. COMPAS isn't being censored from sharing a politically incorrect truth. It's being prevented from ruining people's lives with a really, really lazy application of statistics. I often joke that racial discrimination laws are one of the few examples where "being bad at math" is not just criminal, but unconstitutional. > ...laws are not designed as truth-seeking tools Again, in cases where bias becomes illegal, these models are NOT just being used to seek truth. They're being used to make decisions. You seem to have misconstrued the fundamental thesis of the article. The author isn't calling for death to polite fictions. And there's a concrete example in the article of this point of departure between your perspective and his. Namely, his response to people-as-gorillas was not "fuck you, the math is right". And the crescendo of the piece is "AI is just a tool, not a divine oracle, and there's nothing new under the sun".
TheCoelacanth 9y ago
> This is what happens with loan or crime data sets, and it made plenty of noise in the news recently - but only the kind of noise in which people say it's obviously the algorithm that's broken, because it doesn't fit the "polite fiction" they'd like to believe. The polite fiction would be that it is only the algorithm that's broken when it is actually the country that is broken.
- TeMPOraL 9y ago
  Yes, exactly.