> A surprisingly sticky belief is that a machine learning model merely reflects existing algorithmic bias in the dataset and does not itself contribute to harm.

> A surprisingly sticky belief is that a machine learning model merely reflects existing algorithmic bias in the dataset and does not itself contribute to harm. Why, despite clear evidence to the contrary, does the myth of the impartial model still hold allure for so many within our research community? Algorithms are not impartial, and some design choices are better than others. Recognizing how model design impacts harm opens up new mitigation techniques that are less burdensome than comprehensive data collection. Ah yes the classic "why do people keep insisting on X despite consistent proof of not X", with...zero citations. Well, let's read the paper and see if they make the case. > Even if we are able to label sensitive attributes at scale such as gender and race, algorithms can still leverage proxy variables to reconstruct the forbidden label. Data collection of even a limited number of protected attributes can be onerous. For example, it is hard to align on a standard taxonomy—categories attributed to race or gender are frequently encoded in inconsistent ways across datasets.2 Furthermore, procuring labels for these legally protected attributes is often perceived as intrusive leading to noisy or incomplete labels.3,4 > If we cannot guarantee we have fully addressed bias in data pipeline, the overall harm in a system is a product of the interactions between the data and our model design choices. Here, acknowledging the impact of model design bias can play an important role in curbing harm. Algorithms are not impartial, and some design choices are better than others. Recognizing how model design impacts harm opens up new mitigation techniques that are far less burdensome than comprehensive data collection. > We are well-versed in the connection between function choice and test-set accuracy because objective functions such as cross-entropy or mean squared error reflect our preference to optimize for high test-set accuracy. Standard loss functions do not explicitly encode preferences for other objectives we care about such as algorithmic bias, robustness, compactness, or privacy. However, just because these desiderata are not reflected does not mean they have ceased to exist. Turing award winner Donald Knuth said that computers “do exactly what they are told, no more and no less.” A model can fulfill an objective in many ways, while still violating the spirit of said objective. > Model design choices made to maximize test-set accuracy do not hold static other properties we care about such as robustness and fairness. On the contrary, training a parametric model is akin to having a fixed amount of materials to build a house with. If we decide to use more bricks building a bigger living room, we force the redistribution of the number of bricks available for all other rooms. In the same vein, when we prioritize one objective, whether that be test-set accuracy or additional criteria such as compactness and privacy, we inevitably introduce new trade-offs. This is literally just saying "Well yes all the bias starts out in the data, and so a race/gender-neutral model will encode those features, but in principle you could design a model expressly to avoid that, and since that hasn't been done, the models are racist too". Which is...a certain kind of take. But even if we accept the reasoning, it is not at all at odds with the common understanding of their starting sentence: "A surprisingly sticky belief is that a machine learning model merely reflects existing algorithmic bias in the dataset and does not itself contribute to harm.". Yes, people commonly believe the "bias is in the data". Models are designed to be mirrors, so they reflect whatever bias is in their training data. Yes, it's true you can design a distorted mirror that does not reflect certain attributes. But that would be inserting bias into a model to correct a bias that you found in the original data. It should be very clear that the root source of the bias here is the training data, not the model. We can and should build models that attempt to correct those biases, and we also can and should attempt to de-bias the data itself. But can we stop torturing the meaning of English sentences to support people's preferred narratives? The models are neutral. The data contains the bias. Convolutional neural nets do not inherently work better on white faces. Multi-layer perceptrons do not have beliefs about race and gender. One additional linguistic nitpick: > The belief that model design merely reflects algorithmic bias in the dataset can be partly ascribed to the difficulty of measuring interactions between all the variables we care about. There is no "algorithmic bias in the dataset". That is a contradiction in terms. The bias in the dataset is not "algorithmic". The algorithm is what processes the data. The dataset is just biased, it has not been biased algorithmically.