7 ms·

This article doesn't make any major factual errors, but it's implicating that this is some sort of surprise, and that it's a major impediment, neither or which

This article doesn't make any major factual errors, but it's implicating that this is some sort of surprise, and that it's a major impediment, neither or which hold up. While training on noisy datasets is an interesting problem for many practical use-cases, a few percent label errors in these classic basic datasets (MNIST, CIFAR, ImageNet) really doesn't matter that much. The first two are basically toys to check that your system works at all at this point, and even on ImageNet the best models are now more accurate than the original labels. However, per [1], progress on current ImageNet still correlates with true accuracy. This is in large because we move to harder tests when the easy ones stop working. In part it's also good, because training with label noise forces the use of better-generalizing solutions. The current SOTA is Meta Pseudo Labels[2], which is a particularly clever trick that never even directly exposes the final model to the training data. [1] Are we done with ImageNet? — https://arxiv.org/abs/2006.07159 https://arxiv.org/abs/2006.07159 [2] Meta Pseudo Labels — https://arxiv.org/abs/2003.10580 https://arxiv.org/abs/2003.10580

SpicyLemonZest 6y ago
It's good to hear that experts have strategies to handle the problem, but I think this will definitely be a surprise to most of the article's readers. I'm a relatively AI-interested layman, and I had assumed that these kinds of datasets had more like 99% accuracy.
- Veedrac 6y ago
  Heh, I guess this is a difference in perspectives. ML has some really shoddy datasets, like really bad, worse than you think I mean. > The macro-averaged results show that the ratio of correct samples (“C”) ranges from 24% to 87%, with a large variance across the five audited datasets. Particularly severe problems were found in CCAligned and WikiMatrix, with 44 of the 65 languages that we audited for CCAligned containing under 50% correct sentences, and 19 of the 20 in WikiMatrix. In total, 15 of the 205 language specific samples (7.3%) contained not a single correct sentence. Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets — https://arxiv.org/abs/2103.12028 https://arxiv.org/abs/2103.12028 In the face of stuff like that, a 90+% correct dataset doesn't seem like a big deal.
iudqnolq 6y ago
The article alleges that the error is non-random. There are some specific sorts of errors the humans on mechanical turk made, and the SOTA models get a significant portion of their advantage over earlier models based only on being better at guessing the incorrect labels. To a layperson, this sounds like a different issue to the one you refute. Is this not a big deal?
- Veedrac 6y ago
  Label noise will almost always give you a weaker model, and sometimes it matters whether those are merely random or due to some general confusion (eg. people mistaking crabs for lobsters). But these datasets are best thought of as research tools, not generally directly useful for applied models. Whereas there are major practical issues with even small or subtle errors in, eg., medical or legal datasets for real-world models, a pure research dataset is generally fine as long as a higher score correlates with a smarter model. If MNIST, a nowadays-trivial handwritten digit recognition dataset, had all its ‘1’s labels swapped with ‘3’s labels, it would pretty much fill its role just as well.
currio 6y ago
I think other result from the article (and paper it links to) is with more accurate data simpler models perform better than complicated models. This is a good observation because as the saying as "Garbage in garbage out" or "Model is only as good as data". It is better to focus on better data than spending time or improving models or overfitting complicated models to inaccurate data
- Guest42 6y ago
  Hadn't thought of it, but in my experience the "newer" models require a lot more data, this would them become an impediment because there's a certain likelihood the quality goes down with the increased quantity, i.e. measured differently or under different conditions.

SpicyLemonZest 6y ago

It's good to hear that experts have strategies to handle the problem, but I think this will definitely be a surprise to most of the article's readers. I'm a relatively AI-interested layman, and I had assumed that these kinds of datasets had more like 99% accuracy.

Veedrac 6y ago
Heh, I guess this is a difference in perspectives. ML has some really shoddy datasets, like really bad, worse than you think I mean. > The macro-averaged results show that the ratio of correct samples (“C”) ranges from 24% to 87%, with a large variance across the five audited datasets. Particularly severe problems were found in CCAligned and WikiMatrix, with 44 of the 65 languages that we audited for CCAligned containing under 50% correct sentences, and 19 of the 20 in WikiMatrix. In total, 15 of the 205 language specific samples (7.3%) contained not a single correct sentence. Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets — https://arxiv.org/abs/2103.12028 https://arxiv.org/abs/2103.12028 In the face of stuff like that, a 90+% correct dataset doesn't seem like a big deal.

iudqnolq 6y ago

The article alleges that the error is non-random. There are some specific sorts of errors the humans on mechanical turk made, and the SOTA models get a significant portion of their advantage over earlier models based only on being better at guessing the incorrect labels. To a layperson, this sounds like a different issue to the one you refute. Is this not a big deal?

Veedrac 6y ago
Label noise will almost always give you a weaker model, and sometimes it matters whether those are merely random or due to some general confusion (eg. people mistaking crabs for lobsters). But these datasets are best thought of as research tools, not generally directly useful for applied models. Whereas there are major practical issues with even small or subtle errors in, eg., medical or legal datasets for real-world models, a pure research dataset is generally fine as long as a higher score correlates with a smarter model. If MNIST, a nowadays-trivial handwritten digit recognition dataset, had all its ‘1’s labels swapped with ‘3’s labels, it would pretty much fill its role just as well.

currio 6y ago

I think other result from the article (and paper it links to) is with more accurate data simpler models perform better than complicated models. This is a good observation because as the saying as "Garbage in garbage out" or "Model is only as good as data". It is better to focus on better data than spending time or improving models or overfitting complicated models to inaccurate data

Guest42 6y ago
Hadn't thought of it, but in my experience the "newer" models require a lot more data, this would them become an impediment because there's a certain likelihood the quality goes down with the increased quantity, i.e. measured differently or under different conditions.