4 ms·
You need correct labels to train these models. The point of the article is that some of the most popular benchmark datasets have a significant proportion of inc
by beforeolives 6y ago
You need correct labels to train these models. The point of the article is that some of the most popular benchmark datasets have a significant proportion of incorrect labels.
- sdenton4 6y agoActually, it's ok to have lots of incorrect training labels; the effect is usually just to shift the output probabilities downward. Where you really want/need accurate labels is the eval set, which the article addresses.
- cgn 6y agoI am an author on the paper, and just saw this thread. Thanks for all of your comments. Note this work focuses on errors in test sets, not training labels. Test sets are foundation of benchmarking for machine learning. We study test sets (versus training sets), because training sets are already studied in-depth in confident learning (https://arxiv.org/pdf/1911.00068.pdf https://arxiv.org/pdf/1911.00068.pdf) and because before this paper, no one had studied (at scale, for lots of datasets) if these test sets had errors in the first place (beyond just looking at the well-known erroneous ImageNet dataset, for example) and do these test sets affect benchmarks. The answer is, yes, if the noise prevalence is fairly high (above 10%). The takeaway, is you must benchmark models on clean test sets, regardless of noise in your training set, if you want to know how your model will perform in the real world.
- sdenton4 6y agoThanks for touching back! One of the other comments mentioned that it may be better to treat Imagenet as a multi-label problem. Do you see value in treating the eval set as multi-label instead of single-label? Specifically, it may not be an erroneous label if there's actually multiple things present in the image.