If a good ImageNet model gets 88-90% accuracy on the test set (so, 10%-12% error rate) then if it turns out that 5.8% of the test set "gold standard" labels are

If a good ImageNet model gets 88-90% accuracy on the test set (so, 10%-12% error rate) then if it turns out that 5.8% of the test set "gold standard" labels are actually wrong is a very big deal, it would mean that half of the measured differences aren't actually real but an artifact of the original human labelers' mistakes.