7 ms·

Line 210: > "Gay and heterosexual people were represented in equal numbers." According to Gallup Polling, around 4% of the American population is homosexual.

by ghgr 9y ago

Line 210: > "Gay and heterosexual people were represented in equal numbers." According to Gallup Polling, around 4% of the American population is homosexual. So, let's be generous and say that their classifier has a 80% accuracy given balanced inputs while humans have 60%. Let's sample 100 people of the dataset, 50 of which are homosexual (and the rest heterosexual). If the classifier has 80% accuracy (let's assume false_positive_rate = false_negative_balance since I didn't find information about that), it means that 80 people were correctly classified, 10 heterosexuals were misclassified as homosexuals and 10 homosexuals as heterosexuals. According to the Bayes theorem, "Given a random person, probability of being homosexual assuming that the classifier said so" = "Probability of being homosexual of a random person (the so-called prior)" <times> "Probability of classifier saying it's homosexual, assuming it's indeed homosexual <divided by> "Given a random person, probability of saying it's homosexual". Substituting, we get: P = 0.04 * 0.8 / 0.5 = 0.064 If instead of the classifier we use human "feeling", we get: P=0.04 * 0.6 / 0.5 = 0.048 In summary, 4% of the people are homosexual. If a human thinks someone is homosexual, the probability "increases" to 4.8%. If on the contrary their algorithm believes it's homosexual, the probability of actually being homosexual increases to 6.4%. Not very groundbreaking...

dangom 9y ago
I don't know if I can follow. You say that if their classifier had 100% accuracy, then the probability of being homosexual would increase to 8%? That doesn't make much sense.
- BeniBoy 9y ago
  Look here [1] Very good explanation (though the op's one are too) [1]: https://en.wikipedia.org/wiki/False_positive_paradox https://en.wikipedia.org/wiki/False_positive_paradox
  - dangom 9y ago
    But why does "Given a random person, probability of saying it's homosexual" equal 0.5? I'd say that should be 0.04.
    - BeniBoy 9y ago
      I guess because the algorithm has been trained on 50% straight 50% gay and the 80% figure is actually for telling appart a pair of straight and gay persons, it will say gay half the time.. But this is confusing you are right. The explanation using false positive is clearer. If there is 1000 persons, 40 gay, and the algorithm is 80% accurate, it will say: 960 * 20%= 192 straight people are gay and 40 * 80% = 32 gay person are gay. Not very impressive and shows a very stong gay bias if I can say so
    - palmy 9y ago
      This is the part of his explanation I find confusing too, though I do agree with his general argument: the results aren't really useful when they haven't been tested on a dataset representative of the real distribution. From what I can gather he's basically saying that the "probability of a random face being classified as homosexual" is 0.5. This isn't REALLY true (would have to run the classifier on all possible faces to find this), but that is in fact the "environment" the classifier has been trained within.
      - speedplane 9y ago
        If the test set of images really were 50/50 and the human judges weren't told that, then they were effectively given inaccurate priors, which would obviously reduce their accuracy.
    - deleted 9y ago
      [deleted]
  - jupp0r 9y ago
    There is also https://en.wikipedia.org/wiki/Base_rate_fallacy https://en.wikipedia.org/wiki/Base_rate_fallacy, which describes the common mistakes interpreting these kinds of data.
- plaidfuji 9y ago
  It doesn't make sense, but that's exactly right. If their training sample had been only 4% gay and achieved 100% accuracy, then the true positive rate would be expected to be 1, because the denominator changes to 0.04. It does raise the question of how Bayes' rule applies if your sample is underrepresented, though... say their training set was only 2% gay. Could they achieve a true positive rate of 200%?
- ghgr 9y ago
  I found that confusing, too. I guess that's the best you can expect if you train your model in such an unrealistically balanced dataset. Put it differently. If I had 100% accuracy at detecting depression in a dataset of depressed people, how well will run this "detector" at detecting depression in the real world? Well, being 'd' the incidence of depression: P(being depressed | trivial detector said so) = d * 1.0 / 1.0 = d
speedplane 9y ago
Don't follow your full argument, but I think get the point: the computer is expecting 50/50 hetero and homosexual people, after looking at the training data. However, the human judges likely assumed that the percentage of homosexual people in the study matched that in society, roughly 4%, and would bias their guesses guessing that fewer people were homosexual. Assuming that's true, no wonder the computer did better. I'd bet it would do far worse dealing with a more representative sample. Still, even though it's not a perfect study, it's still impressive it can do as well as it did.
halflings 9y ago
I tried really hard to follow your explanation, but I can't understand why it has to be so complex. Why are you using the accuracy as a conditional likelihood?
Chriky 9y ago
Is there a name for this kind of smug, couple-of-sentences, incorrect, dismissal of proper research? It seems to have infected a lot of the sites I visit, and is always liked or voted to the top. I guess people just love feeling superior.
- plaidfuji 9y ago
  Yes, it's called "peer review".
- n1try 9y ago
  Don't you think these thoughts are justified?
- barryhoodlum 9y ago
  It's not proper research.
deleted 9y ago
[deleted]
danieltillett 9y ago
The homosexual percentage is very likely to be lower than 4%, but getting accurate data is not easy [0]. 0. https://en.wikipedia.org/wiki/LGBT_demographics_of_the_United_States https://en.wikipedia.org/wiki/LGBT_demographics_of_the_Unite...
- Sean1708 9y ago
  > very likely to be lower than 4% Where in the article does it say that? I can see > Studies from several nations, including the U.S., conducted at varying time periods, have produced a statistical range of 1.2[4] to 6.8[6] percent of the adult population identifying as LGBT. but there's no indication of how likely any part of that range is.
  - danieltillett 9y ago
    LGBT != homosexual Just read the whole article. The references cited are also worth reading if you are actually interested in learning about this topic.
kofejnik 9y ago
>> let's assume false_positive_rate = false_negative_balance This assumption is usually wrong when trying to detect rare events (such as CC fraud or straight/gay people). In my own experience, if you train on a prepared 50/50 dataset to some kind of reasonable accuracy, on actual data where classes are more like 96/4 you get a LOT of false positives, thus making the whole thing not very useful
n1try 9y ago
I think the point is that the classifier is not intended to be used to predict sexual orientation in a real-world scenario. It'd be completely useless because it will have way too many false positives. But I think that wasn't the authors' goal anyway. I think they rather wanted to show that there are facial features related to sexual orientation that are "hidden" to humans. And in my opinion the results are fine for that purpose.
ajmurmann 9y ago
That's why it's usually advised to not only look at simple accuracy but also things like precision, recall and F1 score
jpiabrantes 9y ago
Your denominators are wrong. The denominator should be the probability of the classifier being positive: P(True Positive)+P(False Positive) Since you assumed that P(False Positive) = P(False Negative) = (1-P(True Positive))/2 = (1-P(True Negative))/2 In the algorithm case the denominator is 0.8x0.04+0.1x0.96=0.128 In the human case the denominator is 0.6x0.04+0.2x0.96=0.216 In summary, if a human thinks someone is homosexual, the probability increases to 11.1%. If the algorithm makes that call the probability increases to 25%.

by ghgr 9y ago

dangom 9y ago

I don't know if I can follow. You say that if their classifier had 100% accuracy, then the probability of being homosexual would increase to 8%? That doesn't make much sense.

BeniBoy 9y ago
Look here [1] Very good explanation (though the op's one are too) [1]: https://en.wikipedia.org/wiki/False_positive_paradox https://en.wikipedia.org/wiki/False_positive_paradox
- dangom 9y ago
  But why does "Given a random person, probability of saying it's homosexual" equal 0.5? I'd say that should be 0.04.
  - BeniBoy 9y ago
    I guess because the algorithm has been trained on 50% straight 50% gay and the 80% figure is actually for telling appart a pair of straight and gay persons, it will say gay half the time.. But this is confusing you are right. The explanation using false positive is clearer. If there is 1000 persons, 40 gay, and the algorithm is 80% accurate, it will say: 960 * 20%= 192 straight people are gay and 40 * 80% = 32 gay person are gay. Not very impressive and shows a very stong gay bias if I can say so
  - palmy 9y ago
    This is the part of his explanation I find confusing too, though I do agree with his general argument: the results aren't really useful when they haven't been tested on a dataset representative of the real distribution. From what I can gather he's basically saying that the "probability of a random face being classified as homosexual" is 0.5. This isn't REALLY true (would have to run the classifier on all possible faces to find this), but that is in fact the "environment" the classifier has been trained within.
    - speedplane 9y ago
      If the test set of images really were 50/50 and the human judges weren't told that, then they were effectively given inaccurate priors, which would obviously reduce their accuracy.
  - deleted 9y ago
    [deleted]
- jupp0r 9y ago
  There is also https://en.wikipedia.org/wiki/Base_rate_fallacy https://en.wikipedia.org/wiki/Base_rate_fallacy, which describes the common mistakes interpreting these kinds of data.
plaidfuji 9y ago
It doesn't make sense, but that's exactly right. If their training sample had been only 4% gay and achieved 100% accuracy, then the true positive rate would be expected to be 1, because the denominator changes to 0.04. It does raise the question of how Bayes' rule applies if your sample is underrepresented, though... say their training set was only 2% gay. Could they achieve a true positive rate of 200%?
ghgr 9y ago
I found that confusing, too. I guess that's the best you can expect if you train your model in such an unrealistically balanced dataset. Put it differently. If I had 100% accuracy at detecting depression in a dataset of depressed people, how well will run this "detector" at detecting depression in the real world? Well, being 'd' the incidence of depression: P(being depressed | trivial detector said so) = d * 1.0 / 1.0 = d

speedplane 9y ago

Don't follow your full argument, but I think get the point: the computer is expecting 50/50 hetero and homosexual people, after looking at the training data. However, the human judges likely assumed that the percentage of homosexual people in the study matched that in society, roughly 4%, and would bias their guesses guessing that fewer people were homosexual. Assuming that's true, no wonder the computer did better. I'd bet it would do far worse dealing with a more representative sample. Still, even though it's not a perfect study, it's still impressive it can do as well as it did.

halflings 9y ago

I tried really hard to follow your explanation, but I can't understand why it has to be so complex. Why are you using the accuracy as a conditional likelihood?

Chriky 9y ago

Is there a name for this kind of smug, couple-of-sentences, incorrect, dismissal of proper research? It seems to have infected a lot of the sites I visit, and is always liked or voted to the top. I guess people just love feeling superior.

plaidfuji 9y ago
Yes, it's called "peer review".
n1try 9y ago
Don't you think these thoughts are justified?
barryhoodlum 9y ago
It's not proper research.

deleted 9y ago

[deleted]

danieltillett 9y ago

The homosexual percentage is very likely to be lower than 4%, but getting accurate data is not easy [0]. 0. https://en.wikipedia.org/wiki/LGBT_demographics_of_the_United_States https://en.wikipedia.org/wiki/LGBT_demographics_of_the_Unite...

Sean1708 9y ago
> very likely to be lower than 4% Where in the article does it say that? I can see > Studies from several nations, including the U.S., conducted at varying time periods, have produced a statistical range of 1.2[4] to 6.8[6] percent of the adult population identifying as LGBT. but there's no indication of how likely any part of that range is.
- danieltillett 9y ago
  LGBT != homosexual Just read the whole article. The references cited are also worth reading if you are actually interested in learning about this topic.

kofejnik 9y ago

>> let's assume false_positive_rate = false_negative_balance This assumption is usually wrong when trying to detect rare events (such as CC fraud or straight/gay people). In my own experience, if you train on a prepared 50/50 dataset to some kind of reasonable accuracy, on actual data where classes are more like 96/4 you get a LOT of false positives, thus making the whole thing not very useful

n1try 9y ago

I think the point is that the classifier is not intended to be used to predict sexual orientation in a real-world scenario. It'd be completely useless because it will have way too many false positives. But I think that wasn't the authors' goal anyway. I think they rather wanted to show that there are facial features related to sexual orientation that are "hidden" to humans. And in my opinion the results are fine for that purpose.

ajmurmann 9y ago

That's why it's usually advised to not only look at simple accuracy but also things like precision, recall and F1 score

jpiabrantes 9y ago

Your denominators are wrong. The denominator should be the probability of the classifier being positive: P(True Positive)+P(False Positive) Since you assumed that P(False Positive) = P(False Negative) = (1-P(True Positive))/2 = (1-P(True Negative))/2 In the algorithm case the denominator is 0.8x0.04+0.1x0.96=0.128 In the human case the denominator is 0.6x0.04+0.2x0.96=0.216 In summary, if a human thinks someone is homosexual, the probability increases to 11.1%. If the algorithm makes that call the probability increases to 25%.