I think your example is obscuring the issue here a little bit, because you didn't really describe a targeted learning problem: > It is trivial for algorithms t

I think your example is obscuring the issue here a little bit, because you didn't really describe a targeted learning problem: > It is trivial for algorithms to introduce novel racism: consider processing text analysing minority causes. These texts contain statistical associations between politically sensitive terms, negative terms, (and so on). However they are not racist, are not born of any racist project, and most humans reading them would be moved to more positively regard racial terms. The meaning text express consists in what they want their readers to do. It is an activity of coordination (writer/reader); and it here it has no racist aspect. Yes, negative words will associate with black people in such literature. They will also likely associate with white people. That is, emotionally intense language will associate with race in literature related to racial justice issues. That is a perfectly valid inference. What you seem to be implying is that the model will learn the propositional statement "black people are bad", but your example is just about associations, not propositional assertions. Associations are not assignments of quality, they are just that: associations. Such a model would correctly learn such associations and not be in any sense biased, morally or otherwise, because the model is not making decisions or evaluations of people or moral objects. The notion of bias usually talked about in ML is in the context of either: 1. Making statistical decisions (e.g. granting a loan, or predicting criminal recidivism) 2. Providing a service of some kind (e.g. facial recognition in a camera/video, ad targeting) Talking about bias in these cases brings the issue into focus, because there is a morally relevant objective function in these cases. And my point is that in these cases, standard untrained ML models are morally neutral. They come to the table with no preconceptions. In a society without racism, they will not learn racism. In a society without sexism, they will not learn sexism. They only reflect what we feed them. Is it reasonable for me to describe my mirror as biased if I don't like how I look? In theory, I could build another "mirror" that reflects me the way I want. It just wouldn't comport with how I actually look, and we would no longer call that object a mirror. I want to step back for a moment and say that I think we probably agree on the object-level facts here. I believe you can correct the morally biased ML-output problem at the model level or the data level equally well. I'm mostly objecting to the linguistic utility of locating the moral bias equally in each of them. I think that kind of relativism is just not a very useful way to describe things. I think for some reason this idea has formed that saying the bias is in X means we have to correct it in X, but I think it's that view that's mistaken. We can and should correct it wherever we feel is most efficient and effective.