5 ms·

I don't really buy this study. If humans can't get a better accuracy than 55%, I'm convinced this vector is leaking some obvious (maybe high-frequency) informat

I don't really buy this study. If humans can't get a better accuracy than 55%, I'm convinced this vector is leaking some obvious (maybe high-frequency) information that doesn't have anything to do with the face. E.g location of the person.

tweetle_beetle 6y ago
I hope you're right, otherwise we're in for some kind of AI phrenology nightmare in the not too distant future.
- tomrod 6y ago
  We're pretty much there today.
  - mellavora 6y ago
    except for the phrenology part :)
    - tomrod 6y ago
      I have built ML workflow for over a decade now. The amount of hogwash I've seen hocked definitely classifies as phrenology :)
gus_massa 6y ago
The 55% is very fuzzy. They cite https://www.researchgate.net/profile/Konstantin-Tskhay/publication/232255935_Accuracy_in_Categorizing_Perceptually_Ambiguous_Groups_A_Review_and_Meta-Analysis/links/53f6b0220cf2fceacc74d1be/Accuracy-in-Categorizing-Perceptually-Ambiguous-Groups-A-Review-and-Meta-Analysis.pdf https://www.researchgate.net/profile/Konstantin-Tskhay/publi... The only relevant 55% that I can find is: > Allport and Kramer (1946) randomly presented 20 yearbook photographs of Jews and Non-Jews to 223 undergraduate students for 15 s each and asked them to categorize the person in each photograph as Jewish or non-Jewish, or to pass on the trial by indicating a lack of knowledge. The reported median identification for the sample was slightly above chance (55.5%; Allport & Kramer, 1946). Moreover, they found that highly prejudiced people were more accurate at distinguishing Jews from non-Jews So it's not the same set of photos and not the same question.
quasirandom 6y ago
When I read a paper like this I'm looking for four things: (1) the data, (2) the benchmarks, (3) the architecture, (4) the controls/ablation. 1. The data: "We used a sample of 1,085,795 participants from three countries (the U.S., the UK, and Canada; see Table 1) and their self-reported political orientation, age, and gender. Their facial images (one per person) were obtained from their profiles on Facebook or a popular dating website... Facial images were processed using Face++37 to detect faces. Images were cropped around the face-box provided by Face++ (red frame on Fig. 1) and resized to 224 × 224 pixels." 2. The benchmarks: "For example, when asked to distinguish between two faces—one conservative and one liberal—people are correct about 55% of the time." 3. The controls: "What would an algorithm’s accuracy be when distinguishing between faces of people of the same age, gender, and ethnicity? To answer this question, classification accuracies were recomputed using only face pairs of the same age, gender, and ethnicity." A. A complaint: Geography and income are two powerful conditioners. These can leak in so many ways: uncropped background (geography), image color and quality (income), eyeglass shape (geography and income). This study really needs more controls. Geography and income would be a nice start.
- sillysaurusx 6y ago
  Geography and income are two powerful conditioners. These can leak in so many ways: uncropped background (geography), image color and quality (income), eyeglass shape (geography and income). This study really needs more controls. Geography and income would be a nice start. But then the data wouldn't represent the natural world: nature as it is. Raw data is the correct thing to use, because it's what a hypothetical other person would also use if you ran the same experiment yourself.
  - deleted 6y ago
    [deleted]
  - SiempreViernes 6y ago
    Uh, the headline claim is about faces, how does it make sense to then insist that you must leave the background in?
im3w1l 6y ago
It might pick up on scars, tattoos and piercings, testosterone level, diet (in particular weight), bags-under-eyes, glasses.
blowski 6y ago
My political leanings change based on how recently a drank a cup of tea. This obsession with a left-right spectrum is a big part of the problem now. You think teachers are underpaid? Oh obviously you must be a pro-abortion, $15 minimum wage supporting, transgender-rights activist. What's that you say, Christian bakers should be allowed to refuse to bake a cake with a pro-gay message on it? Oh, you must be a gun-toting, pro-life, anti-immigratnt Trump fanatic. This kind of sorting people into simple binary categories, and giving them a "shopping bag" full of opinions they're supposed to hold helps nobody. I'm not sure how this was relevant in anyway to your comment, but I just kinda jumped on my soapbox there.
notahacker 6y ago
Apart from the 55% human accuracy, which apparently comes from a completely unrelated study, the bit that really stands out to me is it reports the accuracy only drops from 72% to 68% when controlling for demographic accuracy in the US (a little more noticeable in the UK). Considering demographics alone gets you 60-90% accuracy on voting intention for many US demographics, it strikes me as extremely odd it would have have so little impact on the model.

tweetle_beetle 6y ago

I hope you're right, otherwise we're in for some kind of AI phrenology nightmare in the not too distant future.

tomrod 6y ago
We're pretty much there today.
- mellavora 6y ago
  except for the phrenology part :)
  - tomrod 6y ago
    I have built ML workflow for over a decade now. The amount of hogwash I've seen hocked definitely classifies as phrenology :)

gus_massa 6y ago

The 55% is very fuzzy. They cite https://www.researchgate.net/profile/Konstantin-Tskhay/publication/232255935_Accuracy_in_Categorizing_Perceptually_Ambiguous_Groups_A_Review_and_Meta-Analysis/links/53f6b0220cf2fceacc74d1be/Accuracy-in-Categorizing-Perceptually-Ambiguous-Groups-A-Review-and-Meta-Analysis.pdf https://www.researchgate.net/profile/Konstantin-Tskhay/publi... The only relevant 55% that I can find is: > Allport and Kramer (1946) randomly presented 20 yearbook photographs of Jews and Non-Jews to 223 undergraduate students for 15 s each and asked them to categorize the person in each photograph as Jewish or non-Jewish, or to pass on the trial by indicating a lack of knowledge. The reported median identification for the sample was slightly above chance (55.5%; Allport & Kramer, 1946). Moreover, they found that highly prejudiced people were more accurate at distinguishing Jews from non-Jews So it's not the same set of photos and not the same question.

quasirandom 6y ago

When I read a paper like this I'm looking for four things: (1) the data, (2) the benchmarks, (3) the architecture, (4) the controls/ablation. 1. The data: "We used a sample of 1,085,795 participants from three countries (the U.S., the UK, and Canada; see Table 1) and their self-reported political orientation, age, and gender. Their facial images (one per person) were obtained from their profiles on Facebook or a popular dating website... Facial images were processed using Face++37 to detect faces. Images were cropped around the face-box provided by Face++ (red frame on Fig. 1) and resized to 224 × 224 pixels." 2. The benchmarks: "For example, when asked to distinguish between two faces—one conservative and one liberal—people are correct about 55% of the time." 3. The controls: "What would an algorithm’s accuracy be when distinguishing between faces of people of the same age, gender, and ethnicity? To answer this question, classification accuracies were recomputed using only face pairs of the same age, gender, and ethnicity." A. A complaint: Geography and income are two powerful conditioners. These can leak in so many ways: uncropped background (geography), image color and quality (income), eyeglass shape (geography and income). This study really needs more controls. Geography and income would be a nice start.

sillysaurusx 6y ago
Geography and income are two powerful conditioners. These can leak in so many ways: uncropped background (geography), image color and quality (income), eyeglass shape (geography and income). This study really needs more controls. Geography and income would be a nice start. But then the data wouldn't represent the natural world: nature as it is. Raw data is the correct thing to use, because it's what a hypothetical other person would also use if you ran the same experiment yourself.
- deleted 6y ago
  [deleted]
- SiempreViernes 6y ago
  Uh, the headline claim is about faces, how does it make sense to then insist that you must leave the background in?

im3w1l 6y ago

It might pick up on scars, tattoos and piercings, testosterone level, diet (in particular weight), bags-under-eyes, glasses.

blowski 6y ago

My political leanings change based on how recently a drank a cup of tea. This obsession with a left-right spectrum is a big part of the problem now. You think teachers are underpaid? Oh obviously you must be a pro-abortion, $15 minimum wage supporting, transgender-rights activist. What's that you say, Christian bakers should be allowed to refuse to bake a cake with a pro-gay message on it? Oh, you must be a gun-toting, pro-life, anti-immigratnt Trump fanatic. This kind of sorting people into simple binary categories, and giving them a "shopping bag" full of opinions they're supposed to hold helps nobody. I'm not sure how this was relevant in anyway to your comment, but I just kinda jumped on my soapbox there.

notahacker 6y ago

Apart from the 55% human accuracy, which apparently comes from a completely unrelated study, the bit that really stands out to me is it reports the accuracy only drops from 72% to 68% when controlling for demographic accuracy in the US (a little more noticeable in the UK). Considering demographics alone gets you 60-90% accuracy on voting intention for many US demographics, it strikes me as extremely odd it would have have so little impact on the model.