4 ms·

I'm a computer vision grad student. A few things concern me about this work. Maybe they're incidental, but I'm not ready to throw my hands up in the air quite y

by gcr 13y ago

I'm a computer vision grad student. A few things concern me about this work. Maybe they're incidental, but I'm not ready to throw my hands up in the air quite yet. - Why wasn't this accepted to CVPR/ECCV/one of the well-established computer vision conferences? I would love to read some of the reviewers' comments about this work before I give further judgment. (If this really is some CVPR preprint, or if it actually is peer-reviewed, I'd feel much better about this.) - Why isn't this work listed on the official curated "LFW Results" page that Erik Learned-Miller maintains? http://vis-www.cs.umass.edu/lfw/results.html http://vis-www.cs.umass.edu/lfw/results.html Is this work so new that Erik hasn't had time to review it yet? - Human performance on LFW is 99.2%, which is higher than what the authors think it is. The performance drops to the (claimed) 97% when we only show humans a tight crop of the face: http://www1.cs.columbia.edu/CAVE/publications/pdfs/Kumar_ICCV09.pdf http://www1.cs.columbia.edu/CAVE/publications/pdfs/Kumar_ICC... They discuss this difference in a paragraph in their conclusion, but I consider it dishonest to use the lower number in the abstract and imply it in the title. In fact, I consider it misleading to put "Surpassing human performance" in the title to begin with, but that's another matter :) - Showing good performance on one dataset (LFW) is certainly not enough to show that this "outperforms humans" in the general case. Getting a state-of-the-art result on LFW these days is like squeezing a drop of water out of a rock; in my opinion, we should turn our attention to harder datasets like GBU now that these "easier" ones are solved. I'm not terribly familiar with Gaussian processes so I'm not sure whether the math works out, but it is a pretty uncommon thing to try in this domain. (Perhaps that's what makes this work interesting, especially since this year seems to be the "Deep Learning is Eating Everyone's Lunch" year) I also wish they describe what final-stage classifier they use for the "GaussianFace as Feature Extractor" model. Often, that's the most important step; it's strange that they didn't compare with POOF/High-dimensional-LBP/Face++'s deep-learned features/any of the other state-of-the-art feature extractors, especially considering how much worse "GaussianFace as a binary classifier" does (93% vs 97% is a huge difference in this dataset) Just my two cents. It definitely demands further exploration. I don't see any obvious mistakes, but I'm not sure why their approach works as well as they claim it does either. Edit: I don't mean to start a witch hunt or anything, but if the authors have the guts to put "Human-level performance" in their title, they're just begging for the community to inspect every detail and point out all the flaws in every minutiae in their work. It's our community's hot button. It's similar to the old adage about how if you want a Linux user to help you, you have to tell them how much Linux sucks. That's where much of my skepticism comes from. The most astounding papers are often the most humble, but "humble" certainly doesn't describe this work.

vlasev 13y ago
For what it's worth, the graph (Figure 4) on page 10 features the label "Human, cropped 97.53%" and later again in the comparison section but not in the abstract. Still, the whole thing raises some red flags!
im3w1l 13y ago
As comparison, the deepface paper also puts human performance at 97.5 %, and also isn't mentioned by Erik Miller. https://www.facebook.com/publications/546316888800776/ https://www.facebook.com/publications/546316888800776/ EDIT: The authors also seem to have presented a paper at ICCV 2013, so they aren't nobodies.
- apu 13y ago
  Generally, new results aren't added to LFW until after the conference takes place. CVPR will happen at the end of June this year.
apu 13y ago
In some sense, the 97% is indeed the more fair number to compare against, assuming that this paper also restricts the algorithms to only see tight crops of the face. Obviously using more than just tight crops will give you more information, but our point in differentiating those cases when measuring human performance was that the way LFW was constructed gives you MUCH more information when using loose crops than "normal" images would. For example, many images of actors are at award shows (the Oscars in particular), and so if you see that kind of background in a pair of images, you can just say "same" and have a very good chance of getting it right. That's what the "inverse crop" experiment shows [1] -- when you block out the face in LFW images, you can still get 94% accuracy! In normal images, however, the background won't normally give you so much information. I do feel somewhat bad that our human verification performance experiment numbers are now being used to create linkbait titles like, "computer algorithms can beat humans," because that's obviously not true (nor have I ever believed it), but in my defense, in 2009 we didn't really think about what the press would do in 2014 when algorithms started saturating on LFW =) [1] http://homes.cs.washington.edu/~neeraj/projects/faceverification/ http://homes.cs.washington.edu/~neeraj/projects/faceverifica...

by gcr 13y ago

vlasev 13y ago

For what it's worth, the graph (Figure 4) on page 10 features the label "Human, cropped 97.53%" and later again in the comparison section but not in the abstract. Still, the whole thing raises some red flags!

im3w1l 13y ago

As comparison, the deepface paper also puts human performance at 97.5 %, and also isn't mentioned by Erik Miller. https://www.facebook.com/publications/546316888800776/ https://www.facebook.com/publications/546316888800776/ EDIT: The authors also seem to have presented a paper at ICCV 2013, so they aren't nobodies.

apu 13y ago
Generally, new results aren't added to LFW until after the conference takes place. CVPR will happen at the end of June this year.

apu 13y ago

In some sense, the 97% is indeed the more fair number to compare against, assuming that this paper also restricts the algorithms to only see tight crops of the face. Obviously using more than just tight crops will give you more information, but our point in differentiating those cases when measuring human performance was that the way LFW was constructed gives you MUCH more information when using loose crops than "normal" images would. For example, many images of actors are at award shows (the Oscars in particular), and so if you see that kind of background in a pair of images, you can just say "same" and have a very good chance of getting it right. That's what the "inverse crop" experiment shows [1] -- when you block out the face in LFW images, you can still get 94% accuracy! In normal images, however, the background won't normally give you so much information. I do feel somewhat bad that our human verification performance experiment numbers are now being used to create linkbait titles like, "computer algorithms can beat humans," because that's obviously not true (nor have I ever believed it), but in my defense, in 2009 we didn't really think about what the press would do in 2014 when algorithms started saturating on LFW =) [1] http://homes.cs.washington.edu/~neeraj/projects/faceverification/ http://homes.cs.washington.edu/~neeraj/projects/faceverifica...