Label noise will almost always give you a weaker model, and sometimes it matters whether those are merely random or due to some general confusion (eg. people mi

Label noise will almost always give you a weaker model, and sometimes it matters whether those are merely random or due to some general confusion (eg. people mistaking crabs for lobsters). But these datasets are best thought of as research tools, not generally directly useful for applied models. Whereas there are major practical issues with even small or subtle errors in, eg., medical or legal datasets for real-world models, a pure research dataset is generally fine as long as a higher score correlates with a smarter model. If MNIST, a nowadays-trivial handwritten digit recognition dataset, had all its ‘1’s labels swapped with ‘3’s labels, it would pretty much fill its role just as well.