3 ms·
The article alleges that the error is non-random. There are some specific sorts of errors the humans on mechanical turk made, and the SOTA models get a signific
by iudqnolq 6y ago
The article alleges that the error is non-random. There are some specific sorts of errors the humans on mechanical turk made, and the SOTA models get a significant portion of their advantage over earlier models based only on being better at guessing the incorrect labels. To a layperson, this sounds like a different issue to the one you refute. Is this not a big deal?
- Veedrac 6y agoLabel noise will almost always give you a weaker model, and sometimes it matters whether those are merely random or due to some general confusion (eg. people mistaking crabs for lobsters). But these datasets are best thought of as research tools, not generally directly useful for applied models. Whereas there are major practical issues with even small or subtle errors in, eg., medical or legal datasets for real-world models, a pure research dataset is generally fine as long as a higher score correlates with a smarter model. If MNIST, a nowadays-trivial handwritten digit recognition dataset, had all its ‘1’s labels swapped with ‘3’s labels, it would pretty much fill its role just as well.