4 ms·
> But the paper is talking about a different kind of bias, say, that a data set of loans has thousands of data points of people born in the first neighborhood a
by darawk 5y ago
> But the paper is talking about a different kind of bias, say, that a data set of loans has thousands of data points of people born in the first neighborhood and tens of data points of people born in the second neighborhood. Even if you were to control for economic status (or perhaps explicitly control for things like ethnicity), an algorithm that performs well on the data set as a whole might perform poorly on the particular subset of people born in the second neighborhood, simply because it has less data. The algorithm might have an acceptable (to its human supervisors) false-positive rate rejecting loans to people born in the first neighborhood, but the exact same model might have a much higher false-positive rate to people born in the second neighborhood.
Right, the paper is talking about models that perform poorly on data clusters with low cardinality in the dataset. This is a problem, but it's an intrinsic problem. We simply don't have enough information about those clusters to make informed judgments.
I think the paper is sort of assuming something is happening in which the model forgets information about the scarce groups to make room for even more information about the dense groups, but I don't think that really makes a lot of sense if you think it through. Most neural networks are information sparse, that is, they have lots of neuronal capacity to spare. They don't need to forget things to learn new things, and if they did, we could solve that problem by simply adding capacity, not by forcing the model to forget things its learned about the dense groups.
I accept that it is in principle possible for things to work the way they're describing, but I think there's very good reasons to believe that they don't, and I think it's pretty telling that this paper contains no math, and no supporting data or experiments to back up this model of statistical learning.
- BoiledCabbage 5y ago> I think the paper is sort of assuming something is happening in which the model forgets information about the scarce groups to make room for even more information about the dense groups, but I don't think that really makes a lot of sense if you think it through Here's an example. A group of 10 people work together and try to decide where to go for lunch every day. 6 love pizza 3 love burgers 1 loves BBQ. Given the choice, the lover of each would pick their food. Imagine every day they need to make a decision, so they build an algorithmic "model" to best evaluate where to go for lunch. They do a vote. The majority 6 wins every day so they eat pizza every day all year. Imagine now instead they choose a different algorithmic model "everyone puts their one preference in a hat, someone picks one and we go there". Now over the course of the year we'd expect ~60% of the time pizza, 30% burgers and 10% BBQ. In both cases there was over/under representation of different groups, but the two different algorithms handled the "bias" differently. Now take that same type of concept and apply it to an ML model and that's similar to what the author is talking about.
- visarga 5y agoThe first algo maximizes happiness (always choosing pizza is the happiest choice in aggregate 0.6*1+0.4*0+0.1*0 = 0.6), while the second maximizes equal chances for everyone (0.6^2+0.3^2+0.1^2 = 0.46). You can't directly compare things that maximize different objectives.