4 ms·

Again: no one cares about CAPTCHA in the deep learning world compared to other more challenging benchmarks. I wouldn’t be surprised that many optimizations coul

by bufo 9y ago

Again: no one cares about CAPTCHA in the deep learning world compared to other more challenging benchmarks. I wouldn’t be surprised that many optimizations could be made with ANY kind of effort put into it. Still waiting for Vicarious to go beyond MNIST and text CPATCHA.

nl 9y ago
This is trueish, but there is more to it than that. It is true for sure that absolute performance on MNIST isn't the most interesting thing in the world. But when introducing a new tool or technique being able to show competitive performance on MNIST is a good way to show that it isn't an entirely useless thing. I'd note that recent Sabour, Frosst and Hinton paper[1] (where they finally got Hinton's capsules to work) spends most of the paper analyzing how it performs on MNIST, and only a short section on other datasets. I assume I don't need to point out that Geoff Hinton does know a little about deep learning, and if he thinks submitting a NIPS paper on MNIST is acceptable in 2017 then I'm not going to argue too hard against it. [1] https://arxiv.org/pdf/1710.09829.pdf https://arxiv.org/pdf/1710.09829.pdf
- real-hacker 9y ago
  +1. I was thinking about the Hinton Capsule paper using MNIST.
- chronice70 9y ago
  And what about the other boys who know a thing or two about deep learning? I don't see any of these people submitting MNIST to NIPS in 2017: Yousha Bengio, Yann LeCun, Ian Goodfellow, Andrew Ng, Ross Girshick, Andrej Karpathy, Pedro Domingos, and the whole DeepMind crew. So yes, submitting experiments on MNIST in 2017 should not be taken seriously.
  - nl 9y ago
    "boys" Not sure what this was supposed to mean? Yes, I think Fei Fei Li's datasets are much better tests than MNIST if that is what you were getting at? I don't see any of these people submitting MNIST to NIPS in 2017 None of them submitted things as entirely new and different as this, either. Having said that, I think my point holds. The completely awesome 2017 "Generalization in Deep Learning" paper[1] was co-authored by Bengio and uses MNIST - because everyone can follow it. Yann LeCun was co-author on the 2017 "Adversarially Regularized Autoencoders for Generating Discrete Structures"[1.5], using MNIST Ian Goodfellow Autoencoder NIPS paper[1] used MNIST as one of its 4 datasets. Yes, it was 2014, but when introducing a new technique using familiar datasets isn't a bad thing. DeepMind's "Bayes by Backprop" (ICML15) used MNIST[2] Another example: the (June 2017) John Langford (Vowpal Wabbit) et. al paper[3] on using Boosting to learn ResNet blocks used MNIST. So yes, I agree there are much better datasets to compare performance on. But to prove something new works, MNIST is a useful dataset. [0] https://arxiv.org/pdf/1710.05468.pdf https://arxiv.org/pdf/1710.05468.pdf [1] http://papers.nips.cc/paper/5423-generative-adversarial-nets http://papers.nips.cc/paper/5423-generative-adversarial-nets [1.5] https://arxiv.org/pdf/1706.04223.pdf https://arxiv.org/pdf/1706.04223.pdf [2] https://deepmind.com/research/publications/weight-uncertainty-neural-networks/ https://deepmind.com/research/publications/weight-uncertaint... [3] https://arxiv.org/pdf/1706.04964.pdf https://arxiv.org/pdf/1706.04964.pdf
singularity2001 9y ago
one of the central points of the blog post was that the problem of CAPTCHAs / general artificial intelligence is NOT solved until letters like these are recognised: https://www.vicarious.com/wp-content/uploads/2017/10/image2017-10-20-2-24-35-1.png https://www.vicarious.com/wp-content/uploads/2017/10/image20... does your network solve/recognise those?
- Filligree 9y ago
  Those are letters?
  - dx034 9y ago
    I had no issue recognising any of those immediately.
    - amelius 9y ago
      Interestingly, the article says: > Neuroscience evidence indicates that contours and surfaces are represented in a factored manner in the brain [8-11], which might be why people have no difficulty imagining a chair made of ice.
    - Filligree 9y ago
      I recognized most of them, but only because I knew what they were supposed to be. I don't think I would have otherwise.

by bufo 9y ago

nl 9y ago

This is trueish, but there is more to it than that. It is true for sure that absolute performance on MNIST isn't the most interesting thing in the world. But when introducing a new tool or technique being able to show competitive performance on MNIST is a good way to show that it isn't an entirely useless thing. I'd note that recent Sabour, Frosst and Hinton paper[1] (where they finally got Hinton's capsules to work) spends most of the paper analyzing how it performs on MNIST, and only a short section on other datasets. I assume I don't need to point out that Geoff Hinton does know a little about deep learning, and if he thinks submitting a NIPS paper on MNIST is acceptable in 2017 then I'm not going to argue too hard against it. [1] https://arxiv.org/pdf/1710.09829.pdf https://arxiv.org/pdf/1710.09829.pdf

real-hacker 9y ago
+1. I was thinking about the Hinton Capsule paper using MNIST.
chronice70 9y ago
And what about the other boys who know a thing or two about deep learning? I don't see any of these people submitting MNIST to NIPS in 2017: Yousha Bengio, Yann LeCun, Ian Goodfellow, Andrew Ng, Ross Girshick, Andrej Karpathy, Pedro Domingos, and the whole DeepMind crew. So yes, submitting experiments on MNIST in 2017 should not be taken seriously.
- nl 9y ago
  "boys" Not sure what this was supposed to mean? Yes, I think Fei Fei Li's datasets are much better tests than MNIST if that is what you were getting at? I don't see any of these people submitting MNIST to NIPS in 2017 None of them submitted things as entirely new and different as this, either. Having said that, I think my point holds. The completely awesome 2017 "Generalization in Deep Learning" paper[1] was co-authored by Bengio and uses MNIST - because everyone can follow it. Yann LeCun was co-author on the 2017 "Adversarially Regularized Autoencoders for Generating Discrete Structures"[1.5], using MNIST Ian Goodfellow Autoencoder NIPS paper[1] used MNIST as one of its 4 datasets. Yes, it was 2014, but when introducing a new technique using familiar datasets isn't a bad thing. DeepMind's "Bayes by Backprop" (ICML15) used MNIST[2] Another example: the (June 2017) John Langford (Vowpal Wabbit) et. al paper[3] on using Boosting to learn ResNet blocks used MNIST. So yes, I agree there are much better datasets to compare performance on. But to prove something new works, MNIST is a useful dataset. [0] https://arxiv.org/pdf/1710.05468.pdf https://arxiv.org/pdf/1710.05468.pdf [1] http://papers.nips.cc/paper/5423-generative-adversarial-nets http://papers.nips.cc/paper/5423-generative-adversarial-nets [1.5] https://arxiv.org/pdf/1706.04223.pdf https://arxiv.org/pdf/1706.04223.pdf [2] https://deepmind.com/research/publications/weight-uncertainty-neural-networks/ https://deepmind.com/research/publications/weight-uncertaint... [3] https://arxiv.org/pdf/1706.04964.pdf https://arxiv.org/pdf/1706.04964.pdf

singularity2001 9y ago

one of the central points of the blog post was that the problem of CAPTCHAs / general artificial intelligence is NOT solved until letters like these are recognised: https://www.vicarious.com/wp-content/uploads/2017/10/image2017-10-20-2-24-35-1.png https://www.vicarious.com/wp-content/uploads/2017/10/image20... does your network solve/recognise those?

Filligree 9y ago
Those are letters?
- dx034 9y ago
  I had no issue recognising any of those immediately.
  - amelius 9y ago
    Interestingly, the article says: > Neuroscience evidence indicates that contours and surfaces are represented in a factored manner in the brain [8-11], which might be why people have no difficulty imagining a chair made of ice.
  - Filligree 9y ago
    I recognized most of them, but only because I knew what they were supposed to be. I don't think I would have otherwise.