8 ms·

I don't understand why this seems to be getting so much attention. There are plenty of small image datasets around, and wide recognition of the issues with MNIS

by jph00 9y ago

I don't understand why this seems to be getting so much attention. There are plenty of small image datasets around, and wide recognition of the issues with MNIST. I see no evidence at all that this particular dataset is better than MNIST. None of the issues they themselves list with MNIST are discussed with relation to their proposed replacement. The benchmarks they provide are entirely useless - sklearn does not claim to be a platform for computer vision models. A quick WRN model gets 96% of this dataset (h/t @ajmooch on Twitter), suggesting that it doesn't deal with the "too easy" issue. The images clearly don't deal with the problem of lack of translation invariance. On the downside, they don't have the same ease of understanding of hand-drawn digits, which is extremely helpful for teaching, debugging, and visualizing.

Q6T46nT668w6i3m 9y ago
> sklearn does not claim to be a platform for computer vision models. There are more than a dozen image classification and segmentation examples on the scikit-learn gallery: http://scikit-learn.org/stable/auto_examples/index.html http://scikit-learn.org/stable/auto_examples/index.html
- jph00 9y ago
  Yes but at this stage they're really toy examples. IIRC the best sklearn result they showed on this benchmark had 20% error, whereas the little WRN network was 4%.
strebler 9y ago
I completely agree. My company does image recognition in fashion and this dataset is both way too easy and not useful. I'd love to see a client with such an easy problem (telling between shoes and bags in inventory images). We regularly get high 90's accuracy on real world images where the system has to auto-crop all on its own. Inventory images are far too easy. What's more, the actual hard cases (like tiny shorts vs tiny skirt, or long blouse vs short dress) would be near impossible to do at such low resolution.
jamesblonde 9y ago
Wide residual networks are not exactly hello-world networks. For teaching (I teach a course on DL), I think this is great. It adds variety for me in terms of an introductory tutorial for DL, and it encourage students to use a GPU as error rates will be high on their laptops.
mrdrozdov 9y ago
IIRC a simple convnet will get higher than 99.0% on MNIST, so maybe it is more difficult than it seems at first glance :)

by jph00 9y ago

Q6T46nT668w6i3m 9y ago

> sklearn does not claim to be a platform for computer vision models. There are more than a dozen image classification and segmentation examples on the scikit-learn gallery: http://scikit-learn.org/stable/auto_examples/index.html http://scikit-learn.org/stable/auto_examples/index.html

jph00 9y ago
Yes but at this stage they're really toy examples. IIRC the best sklearn result they showed on this benchmark had 20% error, whereas the little WRN network was 4%.

strebler 9y ago

I completely agree. My company does image recognition in fashion and this dataset is both way too easy and not useful. I'd love to see a client with such an easy problem (telling between shoes and bags in inventory images). We regularly get high 90's accuracy on real world images where the system has to auto-crop all on its own. Inventory images are far too easy. What's more, the actual hard cases (like tiny shorts vs tiny skirt, or long blouse vs short dress) would be near impossible to do at such low resolution.

jamesblonde 9y ago

Wide residual networks are not exactly hello-world networks. For teaching (I teach a course on DL), I think this is great. It adds variety for me in terms of an introductory tutorial for DL, and it encourage students to use a GPU as error rates will be high on their laptops.

mrdrozdov 9y ago

IIRC a simple convnet will get higher than 99.0% on MNIST, so maybe it is more difficult than it seems at first glance :)