3 ms·

Cathy O'Neill's main concern is that someone might discover something true, useful, and racist / sexist / homophobic / etc. In order to avoid the cognitive dis

Cathy O'Neill's main concern is that someone might discover something true, useful, and racist / sexist / homophobic / etc. In order to avoid the cognitive dissonance it's important to remain appropriately ignorant and not investigate anything where you might discover something "problematic". You can tell she's interested in preventing knowledge from how she handled her job to predict effectiveness of homelessness services - she actively decided to not use particular variables on the grounds they might show a result she was uncomfortable with. That isn't an issue of "being aware of the limitations of machine learning", it's intentional ignorance. http://www.slate.com/articles/technology/future_tense/2016/02/how_to_bring_better_ethics_to_data_science.html http://www.slate.com/articles/technology/future_tense/2016/0...

inimino 10y ago
Yes, "intentional ignorance" has value. Justice is blind. That's why some companies blind gender when screening applications, and why most interviewers aren't likely to ask a candidate about their religion or politics or sexual preferences. Sometimes correlations are self-perpetuating, and it's better to not know about them when making decisions.
- h4nkoslo 10y ago
  Well, it only "has value" if you're worried about discovering the wrong thing. If you think you're going to discover the "correct" thing, then you're apparently morally free to use whatever variables you want.
panglott 10y ago
No. Her concern in that case was that a predictive model that measured the effects of racism on some people would then be used to increase the racial discrimination against them. That is absolutely an ethical question. Her task was not merely to find out facts about the world, but to build systems that would impact the welfare of people in the world—and people in a very precarious position, no less. Race is not a valid biological category; it is a social construct and a salient social category. & as such it remains the site of many unethical uses of power. At times measuring them can reify those injustices; at times it is essential in combating them.
- h4nkoslo 10y ago
  You've essentially restated what I said above. She declined to investigate certain variables because she was worried they would come out with the "wrong" effects (instead she chose to use near-perfect correlates of those variables, like zip code). You can always decline to use a variable after the fact; she decided she didn't want to know the effect in the first place. Suppose the effect had been the opposite of what she suspected?

inimino 10y ago

Yes, "intentional ignorance" has value. Justice is blind. That's why some companies blind gender when screening applications, and why most interviewers aren't likely to ask a candidate about their religion or politics or sexual preferences. Sometimes correlations are self-perpetuating, and it's better to not know about them when making decisions.

h4nkoslo 10y ago
Well, it only "has value" if you're worried about discovering the wrong thing. If you think you're going to discover the "correct" thing, then you're apparently morally free to use whatever variables you want.

panglott 10y ago

No. Her concern in that case was that a predictive model that measured the effects of racism on some people would then be used to increase the racial discrimination against them. That is absolutely an ethical question. Her task was not merely to find out facts about the world, but to build systems that would impact the welfare of people in the world—and people in a very precarious position, no less. Race is not a valid biological category; it is a social construct and a salient social category. & as such it remains the site of many unethical uses of power. At times measuring them can reify those injustices; at times it is essential in combating them.

h4nkoslo 10y ago
You've essentially restated what I said above. She declined to investigate certain variables because she was worried they would come out with the "wrong" effects (instead she chose to use near-perfect correlates of those variables, like zip code). You can always decline to use a variable after the fact; she decided she didn't want to know the effect in the first place. Suppose the effect had been the opposite of what she suspected?