7 ms·

Deep learning is getting all the attention because it gets the best results- If you don't like this, you either have to provide (1) a different measure for resu

by drcode 10y ago

Deep learning is getting all the attention because it gets the best results- If you don't like this, you either have to provide (1) a different measure for results, or (2) give an objective mechanism for evaluating the "worth" of a technique that doesn't involve looking at results. I would still like to hear what the author of the post recommends as a course of action- Maybe he can write a followup post that provides these details to clarify this.

f9us98a 10y ago
It gets the best results for what? Maybe for classification and recognition. But the point of the article is that there is more to computer vision. Stereo, optical flow, geometry, and physics can only be aided by deep learning so much. Another point not mentioned is the computational power required for deep learning. Consider programming the physics for a ball rolling down an incline. You could use (1) the math itself, vs. (2) a neural net. It's clear that the direct math approach could be orders of magnitude faster than neural nets. I wouldn't be surprised if directly coding the physics would achieve 1,000x the performance of a neural net.
- opticalflow 10y ago
  I would posit that a deep learning network that learns to optimize parameters for a complex algorithm outside the convolutional network itself may have immense utility outside the classification problem. Call it a marriage of classic computer vision with deep CNN, or a hybrid approach. I don't think it's a binary decision. A deep CNN can find the optimal parameters (once trained) for a classic CV problem for a given image or video or other dataset, like superresolution, patch-based inpainting, or motion tracking. The training is the most computationally intensive part. As someone with way too many kids, I can testify...
  - argonaut 10y ago
    You seem to be describing hyperparameter optimization, which I seriously doubt CNNs are going to be used for anytime soon.
    - opticalflow 10y ago
      Just curious -- why would you think not?
    - pedrosorio 10y ago
      It has been done already: http://arxiv.org/abs/1502.05700 http://arxiv.org/abs/1502.05700 Edit: Although what they seem to describe is replacing GPs with neural networks in Bayesian optimization which is supposedly more efficient. Since the point of Bayesian optimization is to limit the number of times you have to evaluate a new set of hyperparameters, I am not sure how useful it is to "be able to scale" (i.e. even if maintaining the GP is O(n^3) with the number of evaluations, the costly part should be to evaluate the hyperparameters in the first place) but I haven't read the paper so they may show some high dimensional hyperparameter cases where performing a lot of evaluations pays off.
- AndrewKemendo 10y ago
  The point of all of those though are toward the goal of machine/computer vision which is "understand the physical world through [images]." If training CNNs does it faster/better than having a better understanding of Stereopsis/etc... then who cares?
mattkrause 10y ago
It's currently getting the best results, which isn't exactly the same. Deep learning undeniably works well. It works so well that it has almost completely taken over the field, sucking the oxygen out of other parts of the field. This is a little disconcerting since no one really understands how much of deep learning's success is due to a) the algorithms themselves vs. b) the massive amount of data and computational power being thrown at vision problems. There is something to be said for some "counter-cyclical" efforts to encourage people to keep exploring approaches that aren't just slightly different network topologies, or least to ensure that they're not totally forgotten. That's the message I took away from the article.
- adrianN 10y ago
  For a time neural networks seemed dead and SVMs were sucking the oxygen out of other parts of the field. Then someone figured out how to train deep networks and the tides turned.
  - gok 10y ago
    By "how to train deep networks" are you referring to unsupervised pre-training, or the discovery/development of faster parallel training? (GPUs and distributed ASGD)
    - deleted 10y ago
      [deleted]
  - mattkrause 10y ago
    Sure, though the SVM tide never submerged competing methods quite as deeply as the incoming deep network tide, either in terms of performance or mind-share. Ensemble methods also became popular around the same time, for example. In a way, you could take this as a perfect example of the authors' point: we might have had a lot of these advances sooner if we hadn't let interest in neural networks fall completely off a cliff.
    - 10y ago
nshm 10y ago
It is easy to introduce different measure which will make deep learning fail and which will improve the state of the art and humanness significantly. They are: 1) Learning from small amount of examples, make training database very small with 3-4 pictures by class. 2) Learning across classes. Train on evening pictures and learn to recognize daylight. Train on direct sunlight and learn to recognize on side light. 3) Learning of entities. Train on entity in white and learn to recognize a combined picture. Deep learning community intentionally do not consider these tasks, they prefer to increase the training data size to the limit only huge corporation could afford doing a research.
- PeterisP 10y ago
  Deep learning is currently the best way to solve tasks such as your #1 example - there are great results with learning to identify classes from even a single labeled example; and the proper (only?) way to do this is to use basic feature-extraction layers made by showing the system a bunch of unlabelled images.
deleted 10y ago
[deleted]
thaw13579 10y ago
Benchmarks are indeed valuable, but I think one of the points in the article was that if the field focuses too much on them, research with theoretical depth gets excluded. However, those theoretical papers have value that is hard to quantify--it may take years to see it!
wolfgke 10y ago
> (1) a different measure for results, or (2) give an objective mechanism for evaluating the "worth" of a technique that doesn't involve looking at results. One does science also to understand why something works (i.e. what is the theory behind it). Deep learning gives something that (perhaps) works, but doesn't give an any clue to the question why it works - on what theory does the neural net internally compute in its black box. Based on this criterion one can even argue that deep learning papers should not be considered as science as long as the authors of the deep learning paper make no really hard attempt to present at least a partly understanding of the internal theory that the trained neuronal network executes (i.e. a very detailed interpretation of the weights with ideally falsifiable predictions). This is hard work, I know, but isn't that nearly anything in science?
- webaba 10y ago
  Completely agree with this viewpoint. Explanatory power and reproducibility are two of the most important aspects of a scientific paper. Most deep learning based papers lack both these properties.
  - wolfgke 10y ago
    > Explanatory power and reproducibility are two of the most important aspects of a scientific paper. The reproducability problem should be solvable if the authors would simply release the training data and source code that trains the network (as yould scientific practise would require). Or is there anything else that is necessary to assure reproducability (I'm not that deep into CV papers)?

by drcode 10y ago

f9us98a 10y ago

It gets the best results for what? Maybe for classification and recognition. But the point of the article is that there is more to computer vision. Stereo, optical flow, geometry, and physics can only be aided by deep learning so much. Another point not mentioned is the computational power required for deep learning. Consider programming the physics for a ball rolling down an incline. You could use (1) the math itself, vs. (2) a neural net. It's clear that the direct math approach could be orders of magnitude faster than neural nets. I wouldn't be surprised if directly coding the physics would achieve 1,000x the performance of a neural net.

opticalflow 10y ago
I would posit that a deep learning network that learns to optimize parameters for a complex algorithm outside the convolutional network itself may have immense utility outside the classification problem. Call it a marriage of classic computer vision with deep CNN, or a hybrid approach. I don't think it's a binary decision. A deep CNN can find the optimal parameters (once trained) for a classic CV problem for a given image or video or other dataset, like superresolution, patch-based inpainting, or motion tracking. The training is the most computationally intensive part. As someone with way too many kids, I can testify...
- argonaut 10y ago
  You seem to be describing hyperparameter optimization, which I seriously doubt CNNs are going to be used for anytime soon.
  - opticalflow 10y ago
    Just curious -- why would you think not?
  - pedrosorio 10y ago
    It has been done already: http://arxiv.org/abs/1502.05700 http://arxiv.org/abs/1502.05700 Edit: Although what they seem to describe is replacing GPs with neural networks in Bayesian optimization which is supposedly more efficient. Since the point of Bayesian optimization is to limit the number of times you have to evaluate a new set of hyperparameters, I am not sure how useful it is to "be able to scale" (i.e. even if maintaining the GP is O(n^3) with the number of evaluations, the costly part should be to evaluate the hyperparameters in the first place) but I haven't read the paper so they may show some high dimensional hyperparameter cases where performing a lot of evaluations pays off.
AndrewKemendo 10y ago
The point of all of those though are toward the goal of machine/computer vision which is "understand the physical world through [images]." If training CNNs does it faster/better than having a better understanding of Stereopsis/etc... then who cares?

mattkrause 10y ago

It's currently getting the best results, which isn't exactly the same. Deep learning undeniably works well. It works so well that it has almost completely taken over the field, sucking the oxygen out of other parts of the field. This is a little disconcerting since no one really understands how much of deep learning's success is due to a) the algorithms themselves vs. b) the massive amount of data and computational power being thrown at vision problems. There is something to be said for some "counter-cyclical" efforts to encourage people to keep exploring approaches that aren't just slightly different network topologies, or least to ensure that they're not totally forgotten. That's the message I took away from the article.

adrianN 10y ago
For a time neural networks seemed dead and SVMs were sucking the oxygen out of other parts of the field. Then someone figured out how to train deep networks and the tides turned.
- gok 10y ago
  By "how to train deep networks" are you referring to unsupervised pre-training, or the discovery/development of faster parallel training? (GPUs and distributed ASGD)
  - deleted 10y ago
    [deleted]
- mattkrause 10y ago
  Sure, though the SVM tide never submerged competing methods quite as deeply as the incoming deep network tide, either in terms of performance or mind-share. Ensemble methods also became popular around the same time, for example. In a way, you could take this as a perfect example of the authors' point: we might have had a lot of these advances sooner if we hadn't let interest in neural networks fall completely off a cliff.
  - 10y ago

nshm 10y ago

It is easy to introduce different measure which will make deep learning fail and which will improve the state of the art and humanness significantly. They are: 1) Learning from small amount of examples, make training database very small with 3-4 pictures by class. 2) Learning across classes. Train on evening pictures and learn to recognize daylight. Train on direct sunlight and learn to recognize on side light. 3) Learning of entities. Train on entity in white and learn to recognize a combined picture. Deep learning community intentionally do not consider these tasks, they prefer to increase the training data size to the limit only huge corporation could afford doing a research.

PeterisP 10y ago
Deep learning is currently the best way to solve tasks such as your #1 example - there are great results with learning to identify classes from even a single labeled example; and the proper (only?) way to do this is to use basic feature-extraction layers made by showing the system a bunch of unlabelled images.

deleted 10y ago

[deleted]

thaw13579 10y ago

Benchmarks are indeed valuable, but I think one of the points in the article was that if the field focuses too much on them, research with theoretical depth gets excluded. However, those theoretical papers have value that is hard to quantify--it may take years to see it!

wolfgke 10y ago

> (1) a different measure for results, or (2) give an objective mechanism for evaluating the "worth" of a technique that doesn't involve looking at results. One does science also to understand why something works (i.e. what is the theory behind it). Deep learning gives something that (perhaps) works, but doesn't give an any clue to the question why it works - on what theory does the neural net internally compute in its black box. Based on this criterion one can even argue that deep learning papers should not be considered as science as long as the authors of the deep learning paper make no really hard attempt to present at least a partly understanding of the internal theory that the trained neuronal network executes (i.e. a very detailed interpretation of the weights with ideally falsifiable predictions). This is hard work, I know, but isn't that nearly anything in science?

webaba 10y ago
Completely agree with this viewpoint. Explanatory power and reproducibility are two of the most important aspects of a scientific paper. Most deep learning based papers lack both these properties.
- wolfgke 10y ago
  > Explanatory power and reproducibility are two of the most important aspects of a scientific paper. The reproducability problem should be solvable if the authors would simply release the training data and source code that trains the network (as yould scientific practise would require). Or is there anything else that is necessary to assure reproducability (I'm not that deep into CV papers)?