3 ms·

The phrase "exceeding the accuracy of human raters" is a bit puzzling--if a human misclassifies something whose true class was determined by another human, who

The phrase "exceeding the accuracy of human raters" is a bit puzzling--if a human misclassifies something whose true class was determined by another human, who is in error?

dchichkov 12y ago
Probably exceeding the accuracy of a "single human rater". Note, the definition of a "human rater accuracy" is a bit vague. For example an "Amazon Turk rater" qualifies as a human rater, yet the accuracy of such rater can be easily exceeded by an accuracy of dedicated grad student rater, or an accuracy of a dedicated H1 employee rater. edit: That's my guess from previous (2.5 years back) work in computer vision / imagenet and a quick skim over the article.
davmre 12y ago
The comparison is to humans that see the same data as the algorithm. The "true" labels can be assigned using additional data. For example, if I make a dataset for the task of distinguishing cocker spaniels from springer spaniels, then when I collect the pictures I can be quite confident about the labels I assign (eg by DNA-testing the actual dogs). But when someone else tries to identify the dogs based only on the pictures I took, they might get some of them wrong. The 5.1% "human accuracy" number that gets thrown around for ImageNet comes from Andrej Karpathy attempting to manually label part of the ImageNet test set. He wrote a blog post about the experience (http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/ http://karpathy.github.io/2014/09/02/what-i-learned-from-com...), which has some interesting details. For example, they did actually find that 1.7% of images were genuinely misannotated: that's because ImageNet annotations come from a consensus of Mechanical Turkers, which is less reliable than DNA testing. :-)
karpathy 12y ago
The TLDR is that to work at scale we sacrifice some correctness in the dataset, and what's really being evaluated here is closer to some notion of human agreement. The way the data was collected was not through a 1000-way classification. Instead, we'd query things like "red fox" from a search engine, and then have turkers do the binary task of YES/NO to clean up the results. This is a messy, noisy process but we do our best. In particular, the turkers saying YES/NO to red fox are completely unaware of what other classes are in the dataset - everything they say YES to becomes the "red fox" class in imagenet, in the final discriminative task among the 1000 categories. Also note that if an image contains a whole bunch of things (e.g. see left-most image here http://karpathy.github.io/assets/ilsvrc2.png http://karpathy.github.io/assets/ilsvrc2.png), the turker during data collection would be correct in saying that there is in fact, e.g. a "ruler" in that image, but at test time when you're seeing everything in that image, you only have 5 possible predictions so you have to guess about what you think the "true" label is.
obstinate 12y ago
Well, I don't know how they did it, but you could easily imagine a scenario: - 3 human raters, one computer rater - computer always agrees with 2/3 humans - humans do not always agree with 2/3 humans

The phrase "exceeding the accuracy of human raters" is a bit puzzling--if a human misclassifies something whose true class was determined by another human, who is in error?

dchichkov 12y ago

Probably exceeding the accuracy of a "single human rater". Note, the definition of a "human rater accuracy" is a bit vague. For example an "Amazon Turk rater" qualifies as a human rater, yet the accuracy of such rater can be easily exceeded by an accuracy of dedicated grad student rater, or an accuracy of a dedicated H1 employee rater. edit: That's my guess from previous (2.5 years back) work in computer vision / imagenet and a quick skim over the article.

davmre 12y ago

The comparison is to humans that see the same data as the algorithm. The "true" labels can be assigned using additional data. For example, if I make a dataset for the task of distinguishing cocker spaniels from springer spaniels, then when I collect the pictures I can be quite confident about the labels I assign (eg by DNA-testing the actual dogs). But when someone else tries to identify the dogs based only on the pictures I took, they might get some of them wrong. The 5.1% "human accuracy" number that gets thrown around for ImageNet comes from Andrej Karpathy attempting to manually label part of the ImageNet test set. He wrote a blog post about the experience (http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/ http://karpathy.github.io/2014/09/02/what-i-learned-from-com...), which has some interesting details. For example, they did actually find that 1.7% of images were genuinely misannotated: that's because ImageNet annotations come from a consensus of Mechanical Turkers, which is less reliable than DNA testing. :-)

karpathy 12y ago

The TLDR is that to work at scale we sacrifice some correctness in the dataset, and what's really being evaluated here is closer to some notion of human agreement. The way the data was collected was not through a 1000-way classification. Instead, we'd query things like "red fox" from a search engine, and then have turkers do the binary task of YES/NO to clean up the results. This is a messy, noisy process but we do our best. In particular, the turkers saying YES/NO to red fox are completely unaware of what other classes are in the dataset - everything they say YES to becomes the "red fox" class in imagenet, in the final discriminative task among the 1000 categories. Also note that if an image contains a whole bunch of things (e.g. see left-most image here http://karpathy.github.io/assets/ilsvrc2.png http://karpathy.github.io/assets/ilsvrc2.png), the turker during data collection would be correct in saying that there is in fact, e.g. a "ruler" in that image, but at test time when you're seeing everything in that image, you only have 5 possible predictions so you have to guess about what you think the "true" label is.

obstinate 12y ago

Well, I don't know how they did it, but you could easily imagine a scenario: - 3 human raters, one computer rater - computer always agrees with 2/3 humans - humans do not always agree with 2/3 humans