18 ms·
The "deep gaydar" paper reports that the training data had 50% gay men and 50% straight men and the same for women (for subjects with at least one picture, i.e.
by YeGoblynQueenne 8y ago
The "deep gaydar" paper reports that the training data had 50% gay men and 50% straight men and the same for women (for subjects with at least one picture, i.e. all). This equal distribution is, of course, quite unlike the distribution in the real world. The paper itself quotes a 7% prevalence of homosexuality.
The question is then- if they believed their sources that homosexual men and women are 7% of the population, why did they choose to train their classifier with a radically different distribution. One answer may be that it's much harder to train an accurate classifier for unbalanced classes and that artificially including more of the minority class improves the performance of the classifier on its training and its test set, therefore overall improving its evaluation.
Of course, the resulting model is useless for predictions in the real world, because it violates the central assumption in PAC learning (the computational learning theory behind modern machine learning): that the data used to train a machine learning algorithm has the same distribution as unseen data.
In other words, the "deep gaydar" paper is a pointless exercise in classifier tuning. It proves that, with enough knowledge of how classifiers work, you can squeeze any kind of results you want out of them. But it doesn't tell us anything else.
- dual_basis 8y agoI would disagree, the assumption that the data has the same distribution in your training set as it will in application is not required at all, and in fact it is often better to have more data around the edge of the classification set.
- YeGoblynQueenne 8y agoLike I say, the central assumption in PAC learning is that a data set is drawn from the same distribution as unseen data. Otherwise, there can be no guarantees that the model is capable of representing anything beyond its data set. That is not controversial. I'm surprised at your statement that distributional consistency is "not required at all" and I wonder where it's coming from. I don't understand what you mean by "data around the edge of the classification set".
- BrandoElFollito 8y agoWhy should it be? You present an image and state that this is type A. Then another, with your B, etc. The images are independent, each holds a classification and how many you have from each type does not need to reflect their actual distribution.