Since the amount of speculation in this overall thread is pretty high, I suggest simply having a look at section 3 of [0]. It describes the construction of the

Since the amount of speculation in this overall thread is pretty high, I suggest simply having a look at section 3 of [0]. It describes the construction of the MTurk task and how they quantified the confidence of answers they got. [0] http://www.image-net.org/papers/imagenet_cvpr09.pdf http://www.image-net.org/papers/imagenet_cvpr09.pdf