5 ms·

What's the point of such an incendiary comment? No, you aren't understanding it right. At worst you are offering a deliberately misleading interpretation. Here'

What's the point of such an incendiary comment? No, you aren't understanding it right. At worst you are offering a deliberately misleading interpretation. Here's what the link says: The Boston housing prices dataset has an ethical problem: as investigated in [1], the authors of this dataset engineered a non-invertible variable "B" assuming that racial self-segregation had a positive impact on house prices [2]. Furthermore the goal of the research that led to the creation of this dataset was to study the impact of air quality but it did not give adequate demonstration of the validity of this assumption. The scikit-learn maintainers therefore strongly discourage the use of this dataset unless the purpose of the code is to study and educate about ethical issues in data science and machine learning.

josephcsible 4y ago
The "B" variable measures how integrated a neighborhood is, and that snippet seems to be saying that its existence is the "ethical problem" that led them to purge the dataset. How is any of that different than what I said?
- anigbrowl 4y ago
  [flagged]
  - trinsic2 4y ago
    Stop what?
    - anigbrowl 4y ago
      Expressing incredulity over one piece of context without acknowledging the existence of other parts which modify it. There was a full detailed explanation at the medium blog, and all that was necessary to do was follow the citation given and read it. Repeatedly proclaiming confusion when already in possession of the explanatory material, rather than challenging that explanation, is not a credible posture.
      - trinsic2 4y ago
        If you say so...
- threeseed 4y ago
  Please provide a source that says that "number of blacks in my neighbourhood" is a measure of "neighbourhood integration". It's a ridiculous and offensive premise from my perspective.
  - josephcsible 4y ago
    > Please provide a source that says that "number of blacks in my neighbourhood" is a measure of "neighbourhood integration". It isn't. Bk is "number of blacks in my neighbourhood" as you put it, and the whole point of using B instead of it was so that an all-black neighborhood wouldn't count as more integrated than one with a mix of races.
  - CoastalCoder 4y ago
    > Please provide a source that says that "number of blacks in my neighbourhood" is a measure of "neighbourhood integration". I wonder if the dataset design makes more sense in the context of Boston in particular: [0]. [0] https://en.wikipedia.org/wiki/Boston_desegregation_busing_crisis https://en.wikipedia.org/wiki/Boston_desegregation_busing_cr...
  - CoastalCoder 4y ago
    > It's a ridiculous and offensive premise from my perspective. Can you elaborate on the problem you have with this? (I'm just trying to not guess at your meaning.)
- stevenbedrick 4y ago
  So, the issue is more subtle than that - if it was just a matter of including demographic data, there wouldn't be an issue. The problem is that the "B" column is _not_ measuring how integrated the neighborhood is- it is a transformed value whose calculation begins with data about integration levels, but the details of how and why that transformation is performed are super important. The calculation the original authors did to produce that column rests on a model about the relationship between a neighborhood's level of integration and its property values, and that model's assumptions are frankly racist and also factually incorrect (and were known to be incorrect at the time of its original publication back in the '70s). As a result, if one were to _use_ the "B" column in a model, one would be getting results that at best were wrong and useless, and at worst would make a model that literally encodes broken 70s-era ideas about how real estate and race interact in the US. And the transformation itself is non-invertible, so it's not possible to recover the original values for about 7-8% of the rows in the dataset. The commit diff links to a thorough investigation of the data[1] in which the author takes a crack at linking up the ambiguous rows with the original 1970 Census data that supposedly went in to generating this dataset, and long story short, it looks like the original dataset's authors may have made some errors in their calculations on top of everything else. 1: https://medium.com/@docintangible/racist-data-destruction-113e3eff54a8 https://medium.com/@docintangible/racist-data-destruction-11...
trinsic2 4y ago
How is doing a sanity check incendiary in your view again?
- ffssffss 4y ago
  [flagged]
  - generalizations 4y ago
    > Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
    - ffssffss 4y ago
      Sadly, assumptions of good faith are easily exploited by bad actors (the classic term for this is "just asking questions") but I suppose you're right, I should not have assumed malice.
      - generalizations 4y ago
        Better one one mud slinger than two, for the sake of the community. Guilty of it myself too many times. Cheers.
  - josephcsible 4y ago
    > The non-invertability is part of the problem, and he completely doesn't understand that. I get that invertibility means that you can't fully recover the original racial percentage, e.g., that a 48/52 split and a 78/22 split will both look exactly the same, since (.48-.63)^2 and (.78-.63)^2 are equal. I don't see why that totally taints the entire dataset.
  - europeanguy 4y ago
    > I think he wrote the comment in bad faith. The non-invertability is part of the problem, and he completely doesn't understand that. If he doesn't understand it, then it's not in bad faith. Right?
jimbob45 4y ago
Is it just me or this some horrifically bad English? I have a fairly strong math background and I'm struggling to figure out what the author meant by any of that. As far as I can tell, it's something like, "The author made a bad variable. Also, the goal was to check air quality but the variable was bad." What does the subject being air quality impact have to do with anything there?

josephcsible 4y ago

The "B" variable measures how integrated a neighborhood is, and that snippet seems to be saying that its existence is the "ethical problem" that led them to purge the dataset. How is any of that different than what I said?

anigbrowl 4y ago
[flagged]
- trinsic2 4y ago
  Stop what?
  - anigbrowl 4y ago
    Expressing incredulity over one piece of context without acknowledging the existence of other parts which modify it. There was a full detailed explanation at the medium blog, and all that was necessary to do was follow the citation given and read it. Repeatedly proclaiming confusion when already in possession of the explanatory material, rather than challenging that explanation, is not a credible posture.
    - trinsic2 4y ago
      If you say so...
threeseed 4y ago
Please provide a source that says that "number of blacks in my neighbourhood" is a measure of "neighbourhood integration". It's a ridiculous and offensive premise from my perspective.
- josephcsible 4y ago
  > Please provide a source that says that "number of blacks in my neighbourhood" is a measure of "neighbourhood integration". It isn't. Bk is "number of blacks in my neighbourhood" as you put it, and the whole point of using B instead of it was so that an all-black neighborhood wouldn't count as more integrated than one with a mix of races.
- CoastalCoder 4y ago
  > Please provide a source that says that "number of blacks in my neighbourhood" is a measure of "neighbourhood integration". I wonder if the dataset design makes more sense in the context of Boston in particular: [0]. [0] https://en.wikipedia.org/wiki/Boston_desegregation_busing_crisis https://en.wikipedia.org/wiki/Boston_desegregation_busing_cr...
- CoastalCoder 4y ago
  > It's a ridiculous and offensive premise from my perspective. Can you elaborate on the problem you have with this? (I'm just trying to not guess at your meaning.)
stevenbedrick 4y ago
So, the issue is more subtle than that - if it was just a matter of including demographic data, there wouldn't be an issue. The problem is that the "B" column is _not_ measuring how integrated the neighborhood is- it is a transformed value whose calculation begins with data about integration levels, but the details of how and why that transformation is performed are super important. The calculation the original authors did to produce that column rests on a model about the relationship between a neighborhood's level of integration and its property values, and that model's assumptions are frankly racist and also factually incorrect (and were known to be incorrect at the time of its original publication back in the '70s). As a result, if one were to _use_ the "B" column in a model, one would be getting results that at best were wrong and useless, and at worst would make a model that literally encodes broken 70s-era ideas about how real estate and race interact in the US. And the transformation itself is non-invertible, so it's not possible to recover the original values for about 7-8% of the rows in the dataset. The commit diff links to a thorough investigation of the data[1] in which the author takes a crack at linking up the ambiguous rows with the original 1970 Census data that supposedly went in to generating this dataset, and long story short, it looks like the original dataset's authors may have made some errors in their calculations on top of everything else. 1: https://medium.com/@docintangible/racist-data-destruction-113e3eff54a8 https://medium.com/@docintangible/racist-data-destruction-11...

trinsic2 4y ago

How is doing a sanity check incendiary in your view again?

ffssffss 4y ago
[flagged]
- generalizations 4y ago
  > Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - ffssffss 4y ago
    Sadly, assumptions of good faith are easily exploited by bad actors (the classic term for this is "just asking questions") but I suppose you're right, I should not have assumed malice.
    - generalizations 4y ago
      Better one one mud slinger than two, for the sake of the community. Guilty of it myself too many times. Cheers.
- josephcsible 4y ago
  > The non-invertability is part of the problem, and he completely doesn't understand that. I get that invertibility means that you can't fully recover the original racial percentage, e.g., that a 48/52 split and a 78/22 split will both look exactly the same, since (.48-.63)^2 and (.78-.63)^2 are equal. I don't see why that totally taints the entire dataset.
- europeanguy 4y ago
  > I think he wrote the comment in bad faith. The non-invertability is part of the problem, and he completely doesn't understand that. If he doesn't understand it, then it's not in bad faith. Right?

jimbob45 4y ago

Is it just me or this some horrifically bad English? I have a fairly strong math background and I'm struggling to figure out what the author meant by any of that. As far as I can tell, it's something like, "The author made a bad variable. Also, the goal was to check air quality but the variable was bad." What does the subject being air quality impact have to do with anything there?