3 ms·
This criticism is addressed right in the beginning: The foundational discoveries in genetics centered on far simpler organisms such as peas, molds, fruit flies
by p1esk 4y ago
This criticism is addressed right in the beginning:
The foundational discoveries in genetics centered on far simpler organisms such as peas, molds, fruit flies, and mice. To this day, biologists use these simpler organisms as genetic “minimal working examples” in order to save time, energy, and money. A well-designed experiment with Drosophilia, such as Feany and Bender (2000), can teach us an astonishing amount about humans.
The deep learning analogue of Drosophilia is the MNIST dataset. A large number of deep learning innovations including dropout, Adam, convolutional networks, generative adversarial networks, and variational autoencoders began life as MNIST experiments. Once these innovations proved themselves on small-scale experiments, scientists found ways to scale them to larger and more impactful applications.
- in3d 4y agoIt might “address” the criticism but it doesn’t rebut it. It’s absolutely true that MNIST can be misleading and the results on it might not be representative.
- p1esk 4y agoOkay, what would you rather start your experiments on to do a quick sanity check?
- stared 4y agoAgain, MNIST is not Drosophilia of deep learning. For deep learning, it is a trivial. And often misleading, a noticed by François Chollet: > Many good ideas will not work well on MNIST (e.g. batch norm). Inversely many bad ideas may work on MNIST and no[t] transfer to real [computer vision]. It might be considered Drosophilia of machine learning, though. Take a look at this beautiful table of the results of various classifiers http://yann.lecun.com/exdb/mnist/ http://yann.lecun.com/exdb/mnist/.
- p1esk 4y agoMany good ideas will not work well on MNIST (e.g. batch norm). Inversely many bad ideas may work on MNIST and no[t] transfer to real [computer vision]. Unless your dataset is more similar to mnist than to “real computer vision” (eg galaxies or piano rolls) and you still want to use deep learning to classify it.
- stared 4y agoThen, could you give an example of a concrete MNIST-like dataset you worked with? Galaxies are not like such (more diverse, more noise, you don't start with centered images, usually more data - in size and channels).
- p1esk 4y agoAll similarly sparse data samples would suffer from the bachnorm issue. I don’t remember if I tried a convnet with batchnorm on galaxy classification but I did try it on piano rolls - it was bad - precisely because of batchnorm, and had I first tried the same model on mnist I would have caught the issue much faster (I tested it on cifar). I suspect a chess position evaluation would suffer from batchnorm just as much, if the intermediate feature maps remain sparse.