4 ms·

No. The dataset [1] defines B as: 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town And not sure how anyone can argue the dataset is worthy of be

No. The dataset [1] defines B as: 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town And not sure how anyone can argue the dataset is worthy of being included. It is pretty offensive and misguided at minimum to argue that having more black people in your neighbourhood will depress housing prices. And for it to be solely because they are black and not to do with a range of other factors e.g. socio-economic. [1] http://lib.stat.cmu.edu/datasets/boston http://lib.stat.cmu.edu/datasets/boston

josephcsible 4y ago
Isn't that a measure of how integrated a neighborhood is? And even accepting for the sake of argument that such a variable is evil, why not just exclude it instead of ditching the whole dataset?
- deleted 4y ago
  [deleted]
- threeseed 4y ago
  a) No. If 50% of the community is African-Americans and live on one half and everyone else on the other then it would be exactly integrated. Except of course it isn't. b) It is not scikit-learn's responsibility to alter third party datasets.
  - CoastalCoder 4y ago
    > a) No. If 50% of the community is African-Americans and live on one half and everyone else on the other then it would be exactly integrated. Except of course it isn't. IIUC, you're arguing that "whole-town" level aggregation is misleading. So if we get more granular, we could do it my neighborhood, street, building, apartment/car/shelter, bedroom, bed, bed @ time of day, etc. Any one of those aggregation levels could hide interesting distinctions that could be made if only the data were reported with even more granularity. So are you arguing against aggregation in general? Or just whole-town aggregation specifically?
mdcds 4y ago
data is just that, not an argument in itself. how you use it to disprove something is up to you.
- guipsp 4y ago
  Except that this is not just data! The feature is just overall bad.
  - postalrat 4y ago
    More bad than evil?
    - guipsp 4y ago
      Even if it wasn't evil, it's still bad is my point.
mateo411 4y ago
How did they arrive at .63? It's seems like a weird choice. If there are 62% or 64% then the feature will yield the same value. I think it would make more sense just to include percentage of households where at least one member of the household has a certain ethnicity. I don't think it's a offensive to analyze demographic information in the aggregate. In fact this happens all the time.
- josephcsible 4y ago
  I wonder if that was the overall proportion for the entire surrounding area at the time. If so, then B would be a measure of how different the racial makeup of a given subset is from the entire area.
- deleted 4y ago
  [deleted]
Yajirobe 4y ago
Data can show a correlation. What the underlying causation is is not what the dataset aims to answer.
kortilla 4y ago
Offensive to who? It sounds like a great way to compare racism in different areas. The more it correlates with a lower price, the more race issues.
version_five 4y ago
It is pretty offensive and misguided at minimum to argue that having more black people in your neighbourhood will depress housing prices. I think that's the wrong lens to look at this through. I'm happy to concede your statement about it being offensive is true (although I think from a purely statistical perspective, correlations with poverty, etc probably make the assumption correct. Before 2015 or so when we all lost it, it would only be racist to say there was a causal relationship between race and price, not a correlation). Anyway, that's all an aside. It's the purging of a dataset, a toy dataset in this context, for a reason of political correctness, that I don't support. If you look hard enough at anything, you can probably find a way to call it racist or some similar slur. If we start applying this lens to tools like scikit learn, we go down a path I don't agree with, that's completely performative in terms of actually addressing any wrongs, and is a continuing distraction from what could be useful work. Debating if and how racist this is is immaterial imo to whether or not we should erase everything doesn't align with modern hypersensitivity about political correctness
- micromacrofoot 4y ago
  You're missing the fact that there are a lot of reasons the data itself is actually bad... another commenter shared a reasonable analysis: https://medium.com/@docintangible/racist-data-destruction-113e3eff54a8 https://medium.com/@docintangible/racist-data-destruction-11... So even if you want to ignore the fact that the data was outright used to discriminate in the past, the data itself is actually flawed in several ways...
  - skfingngihg 4y ago
    I’m sure we can find flaws in any number of the included data sets, but we’re not excluding those, right? Why are people singling out this one dataset?
    - micromacrofoot 4y ago
      because this one is flawed and comes with some racist presumptions made by the people that originally correlated the data (the presumption being that black people moving into a neighborhood decreases property values... a particularly white-biased correlation without causation)
thw09j9m 4y ago
> offensive and misguided The data is the data. The data isn't suggesting that "having more black people in your neighbourhood will depress housing prices." That's your take on what a racist causal interpretation would look like. The correlation is very real and turning a blind eye to it is worse: https://www.brookings.edu/testimonies/how-racial-disparities-in-home-prices-reveal-widespread-discrimination/ https://www.brookings.edu/testimonies/how-racial-disparities...
- micromacrofoot 4y ago
  That's actually a racist causal interpretation made about the data when it was originally analyzed: > At low to moderate levels of B, an increase in B should have a negative influence on housing value if Blacks are regarded as undesirable neighbors by Whites.
  - RobotToaster 4y ago
    Isn't that saying that lower house prices are caused by racism?
    - micromacrofoot 4y ago
      In an incredible whitewashed way that's very familiar to anyone that lived during the Jim Crow era, yes. Note that it doesn't say that white people leave because they're intolerant of black neighbors, but paints black neighbors as undesirable. Either way, it's a dramatic oversimplification. Black people are also significantly poorer than white people, so it's also likely that black people can only afford to move into a neighborhood as property values decrease... yet the people who collated this data chose to make a different assumption.
- micromacrofoot 4y ago
  That's actually a racist causal interpretation made about the data when it was originally compiled and analyzed... which raises a lot of alarm bells: > At low to moderate levels of B, an increase in B should have a negative influence on housing value if Blacks are regarded as undesirable neighbors by Whites.
sidlls 4y ago
But it’s a fact that—due to the very racism you’re pointing out—house prices tend to be lower the more minorities there are. That’s in part because of the racist policies of banks and the real estate industry. Ignoring it doesn’t do anyone a service. Now, whether it’s used/included responsibly in this dataset is another matter entirely.

josephcsible 4y ago

Isn't that a measure of how integrated a neighborhood is? And even accepting for the sake of argument that such a variable is evil, why not just exclude it instead of ditching the whole dataset?

deleted 4y ago
[deleted]
threeseed 4y ago
a) No. If 50% of the community is African-Americans and live on one half and everyone else on the other then it would be exactly integrated. Except of course it isn't. b) It is not scikit-learn's responsibility to alter third party datasets.
- CoastalCoder 4y ago
  > a) No. If 50% of the community is African-Americans and live on one half and everyone else on the other then it would be exactly integrated. Except of course it isn't. IIUC, you're arguing that "whole-town" level aggregation is misleading. So if we get more granular, we could do it my neighborhood, street, building, apartment/car/shelter, bedroom, bed, bed @ time of day, etc. Any one of those aggregation levels could hide interesting distinctions that could be made if only the data were reported with even more granularity. So are you arguing against aggregation in general? Or just whole-town aggregation specifically?

mdcds 4y ago

data is just that, not an argument in itself. how you use it to disprove something is up to you.

guipsp 4y ago
Except that this is not just data! The feature is just overall bad.
- postalrat 4y ago
  More bad than evil?
  - guipsp 4y ago
    Even if it wasn't evil, it's still bad is my point.

mateo411 4y ago

How did they arrive at .63? It's seems like a weird choice. If there are 62% or 64% then the feature will yield the same value. I think it would make more sense just to include percentage of households where at least one member of the household has a certain ethnicity. I don't think it's a offensive to analyze demographic information in the aggregate. In fact this happens all the time.

josephcsible 4y ago
I wonder if that was the overall proportion for the entire surrounding area at the time. If so, then B would be a measure of how different the racial makeup of a given subset is from the entire area.
deleted 4y ago
[deleted]

Yajirobe 4y ago

Data can show a correlation. What the underlying causation is is not what the dataset aims to answer.

kortilla 4y ago

Offensive to who? It sounds like a great way to compare racism in different areas. The more it correlates with a lower price, the more race issues.

version_five 4y ago

It is pretty offensive and misguided at minimum to argue that having more black people in your neighbourhood will depress housing prices. I think that's the wrong lens to look at this through. I'm happy to concede your statement about it being offensive is true (although I think from a purely statistical perspective, correlations with poverty, etc probably make the assumption correct. Before 2015 or so when we all lost it, it would only be racist to say there was a causal relationship between race and price, not a correlation). Anyway, that's all an aside. It's the purging of a dataset, a toy dataset in this context, for a reason of political correctness, that I don't support. If you look hard enough at anything, you can probably find a way to call it racist or some similar slur. If we start applying this lens to tools like scikit learn, we go down a path I don't agree with, that's completely performative in terms of actually addressing any wrongs, and is a continuing distraction from what could be useful work. Debating if and how racist this is is immaterial imo to whether or not we should erase everything doesn't align with modern hypersensitivity about political correctness

micromacrofoot 4y ago
You're missing the fact that there are a lot of reasons the data itself is actually bad... another commenter shared a reasonable analysis: https://medium.com/@docintangible/racist-data-destruction-113e3eff54a8 https://medium.com/@docintangible/racist-data-destruction-11... So even if you want to ignore the fact that the data was outright used to discriminate in the past, the data itself is actually flawed in several ways...
- skfingngihg 4y ago
  I’m sure we can find flaws in any number of the included data sets, but we’re not excluding those, right? Why are people singling out this one dataset?
  - micromacrofoot 4y ago
    because this one is flawed and comes with some racist presumptions made by the people that originally correlated the data (the presumption being that black people moving into a neighborhood decreases property values... a particularly white-biased correlation without causation)

thw09j9m 4y ago

> offensive and misguided The data is the data. The data isn't suggesting that "having more black people in your neighbourhood will depress housing prices." That's your take on what a racist causal interpretation would look like. The correlation is very real and turning a blind eye to it is worse: https://www.brookings.edu/testimonies/how-racial-disparities-in-home-prices-reveal-widespread-discrimination/ https://www.brookings.edu/testimonies/how-racial-disparities...

micromacrofoot 4y ago
That's actually a racist causal interpretation made about the data when it was originally analyzed: > At low to moderate levels of B, an increase in B should have a negative influence on housing value if Blacks are regarded as undesirable neighbors by Whites.
- RobotToaster 4y ago
  Isn't that saying that lower house prices are caused by racism?
  - micromacrofoot 4y ago
    In an incredible whitewashed way that's very familiar to anyone that lived during the Jim Crow era, yes. Note that it doesn't say that white people leave because they're intolerant of black neighbors, but paints black neighbors as undesirable. Either way, it's a dramatic oversimplification. Black people are also significantly poorer than white people, so it's also likely that black people can only afford to move into a neighborhood as property values decrease... yet the people who collated this data chose to make a different assumption.
micromacrofoot 4y ago
That's actually a racist causal interpretation made about the data when it was originally compiled and analyzed... which raises a lot of alarm bells: > At low to moderate levels of B, an increase in B should have a negative influence on housing value if Blacks are regarded as undesirable neighbors by Whites.

sidlls 4y ago

But it’s a fact that—due to the very racism you’re pointing out—house prices tend to be lower the more minorities there are. That’s in part because of the racist policies of banks and the real estate industry. Ignoring it doesn’t do anyone a service. Now, whether it’s used/included responsibly in this dataset is another matter entirely.