The comparison is to humans that see the same data as the algorithm. The "true" labels can be assigned using additional data. For example, if I make a dataset

The comparison is to humans that see the same data as the algorithm. The "true" labels can be assigned using additional data. For example, if I make a dataset for the task of distinguishing cocker spaniels from springer spaniels, then when I collect the pictures I can be quite confident about the labels I assign (eg by DNA-testing the actual dogs). But when someone else tries to identify the dogs based only on the pictures I took, they might get some of them wrong. The 5.1% "human accuracy" number that gets thrown around for ImageNet comes from Andrej Karpathy attempting to manually label part of the ImageNet test set. He wrote a blog post about the experience (http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/ http://karpathy.github.io/2014/09/02/what-i-learned-from-com...), which has some interesting details. For example, they did actually find that 1.7% of images were genuinely misannotated: that's because ImageNet annotations come from a consensus of Mechanical Turkers, which is less reliable than DNA testing. :-)