4 ms·

There are dozens of papers "explaining" neural networks, using basic math, highly abstract math and even theoretical quantum physics[1] models. I don't see any

There are dozens of papers "explaining" neural networks, using basic math, highly abstract math and even theoretical quantum physics[1] models. I don't see any of them used in constructing, improving or interpreting results of actual neural networks[0]. Moreover, while most of them have math that would take me weeks to fully understand, they rarely make concrete, testable predictions. Most of the recent advances in ANN learning came from using more data and using different types of layers. A real theory of neural nets would explain why architecture A works better than architecture B for a particular problem. Or why a certain architecture plateaus after working with X samples, while other can learn from many more. The way I understand it, the core problem of understanding deep learning is that it does not scale down. You can make a simple framework of 3-4 neurons and observe how it works. Then you make it bigger and it works differently. No one properly explained why this happens. People who do applications seem to just state their intuitions and then throw a couple formulas on top to make it seem scientific. There is a great essay on that by Piekniewski[2]. No research I've seen lists all the parameters/architectures/datasets they tried that didn't work. [0] - Maybe with the exception of the paper on neural networks as ODEs. [1] - http://neurophysics.huji.ac.il/sites/default/files/PhysRevA.32.1007.pdf http://neurophysics.huji.ac.il/sites/default/files/PhysRevA.... [2] - https://blog.piekniewski.info/2018/10/13/deep-learning-the-why-question/ https://blog.piekniewski.info/2018/10/13/deep-learning-the-w...

deleted 8y ago
[deleted]
dnautics 8y ago
Really? Can you honestly say with a straight face that there is no theory behind why convnets are the right choice? Or no theory behind how an LSTM is structured?
- zazen 8y ago
  There is plenty of room to be dissatisfied with the current state of deep learning theory without making absolute claims that there is "no theory" whatsoever, which GP clearly isn't. If you wish to challenge GP's comment, you could do so constructively by discussing any specific theoretical contributions you feel GP may have overlooked.
  - dnautics 8y ago
    > You can make a simple framework of 3-4 neurons and observe how it works. Then you make it bigger and it works differently. No one properly explained why this happens.
    - minkzilla 8y ago
      Can you cite an explanation? I'd love to read one.
      - dnautics 8y ago
        Sure. you can watch one that's in very easy to understand plain english. prereq: https://www.youtube.com/watch?v=C_zFhWdM4ic https://www.youtube.com/watch?v=C_zFhWdM4ic substance: https://www.youtube.com/watch?v=py5byOOHZM8 https://www.youtube.com/watch?v=py5byOOHZM8 I mean what is the difference between like using 50 layers versus 75 layers? Ok, yeah we don't know, but to presume that we don't understand anything beyond 3-4 neurons is just nonsense. By comparison. Go look at any synthetic chemistry paper. There's going to be crazy procedures. One time Phil Baran used Xenon Tetrafluoride as an oxidizing agent. And yeah, for some reason none of the other standard oxidizing agents worked. We don't know why. Hell it could have been that the grad student had bad hands with one of the standard ones. We can make a SWAG an to why XeF4 worked. But we'd be fooling ourselves to claim "we know", so we don't say we do. Does that discredit synthetic chemistry as an enterprise? Does that mean synthetic chemistry is not backed by theory?
        zazen 8y ago
        That video is an introduction to the concept of CNNs for beginners. It explains in informal terms the structure and mechanics of a CNN, and probably gets around to discussing why it might work in very broad terms. This is extremely far from what academically inclined people are talking about when they are using the word "theory". People are hoping to see, ultimately, mathematical proofs that neural networks can e.g. efficiently learn target functions given a certain amount of training data and a certain number of trainable parameters. To address your chemistry analogy: no, it doesn't discredit synthetic chemistry as an enterprise, it just means there is a gap in the theory. Similarly, the people bemoaning gaps in the deep learning theory aren't trying to discredit deep learning as an enterprise: the practical successes are very real; the theory is just lagging, as it often has in application-led fields.
- gambler 8y ago
  >Can you honestly say with a straight face that there is no theory behind why convnets are the right choice? There is theory of convnets, but it's more of a post-factum analysis, really, and I'm not sure it's behind much of anything at this point. Researchers use convnets because they work in ways that work for them. Theory: 109 citations. https://www.semanticscholar.org/paper/A-Mathematical-Theory-of-Deep-Convolutional-Neural-Wiatowski-Bx00F6lcskei/4043604116f90200c3b7cc7573a133349a24bb78?navId=citing-papers https://www.semanticscholar.org/paper/A-Mathematical-Theory-... Practice: 42,551 citations. https://www.semanticscholar.org/paper/ImageNet-Classification-with-Deep-Convolutional-Krizhevsky-Sutskever/2315fc6c2c0c4abd2443e26a26e7bb86df8e24cc?navId=citing-papers https://www.semanticscholar.org/paper/ImageNet-Classificatio...
  - dnautics 8y ago
    > Researchers use convnets because they work in ways that work for them. What about the first person who made a convnet? This is basically like saying there is no understanding of the wheel since there are relatively few citations on the theory of the wheel relative to papers that use wheels.
    - zazen 8y ago
      The first person who made a convnet had an intuition. That is to say, an informal heuristic that gave him reasonable grounds to suppose it might work. This is not "theory". You do not understand what "theory" means in an academic context.

deleted 8y ago

[deleted]

dnautics 8y ago

Really? Can you honestly say with a straight face that there is no theory behind why convnets are the right choice? Or no theory behind how an LSTM is structured?

zazen 8y ago
There is plenty of room to be dissatisfied with the current state of deep learning theory without making absolute claims that there is "no theory" whatsoever, which GP clearly isn't. If you wish to challenge GP's comment, you could do so constructively by discussing any specific theoretical contributions you feel GP may have overlooked.
- dnautics 8y ago
  > You can make a simple framework of 3-4 neurons and observe how it works. Then you make it bigger and it works differently. No one properly explained why this happens.
  - minkzilla 8y ago
    Can you cite an explanation? I'd love to read one.
    - dnautics 8y ago
      Sure. you can watch one that's in very easy to understand plain english. prereq: https://www.youtube.com/watch?v=C_zFhWdM4ic https://www.youtube.com/watch?v=C_zFhWdM4ic substance: https://www.youtube.com/watch?v=py5byOOHZM8 https://www.youtube.com/watch?v=py5byOOHZM8 I mean what is the difference between like using 50 layers versus 75 layers? Ok, yeah we don't know, but to presume that we don't understand anything beyond 3-4 neurons is just nonsense. By comparison. Go look at any synthetic chemistry paper. There's going to be crazy procedures. One time Phil Baran used Xenon Tetrafluoride as an oxidizing agent. And yeah, for some reason none of the other standard oxidizing agents worked. We don't know why. Hell it could have been that the grad student had bad hands with one of the standard ones. We can make a SWAG an to why XeF4 worked. But we'd be fooling ourselves to claim "we know", so we don't say we do. Does that discredit synthetic chemistry as an enterprise? Does that mean synthetic chemistry is not backed by theory?
      - zazen 8y ago
        That video is an introduction to the concept of CNNs for beginners. It explains in informal terms the structure and mechanics of a CNN, and probably gets around to discussing why it might work in very broad terms. This is extremely far from what academically inclined people are talking about when they are using the word "theory". People are hoping to see, ultimately, mathematical proofs that neural networks can e.g. efficiently learn target functions given a certain amount of training data and a certain number of trainable parameters. To address your chemistry analogy: no, it doesn't discredit synthetic chemistry as an enterprise, it just means there is a gap in the theory. Similarly, the people bemoaning gaps in the deep learning theory aren't trying to discredit deep learning as an enterprise: the practical successes are very real; the theory is just lagging, as it often has in application-led fields.
gambler 8y ago
>Can you honestly say with a straight face that there is no theory behind why convnets are the right choice? There is theory of convnets, but it's more of a post-factum analysis, really, and I'm not sure it's behind much of anything at this point. Researchers use convnets because they work in ways that work for them. Theory: 109 citations. https://www.semanticscholar.org/paper/A-Mathematical-Theory-of-Deep-Convolutional-Neural-Wiatowski-Bx00F6lcskei/4043604116f90200c3b7cc7573a133349a24bb78?navId=citing-papers https://www.semanticscholar.org/paper/A-Mathematical-Theory-... Practice: 42,551 citations. https://www.semanticscholar.org/paper/ImageNet-Classification-with-Deep-Convolutional-Krizhevsky-Sutskever/2315fc6c2c0c4abd2443e26a26e7bb86df8e24cc?navId=citing-papers https://www.semanticscholar.org/paper/ImageNet-Classificatio...
- dnautics 8y ago
  > Researchers use convnets because they work in ways that work for them. What about the first person who made a convnet? This is basically like saying there is no understanding of the wheel since there are relatively few citations on the theory of the wheel relative to papers that use wheels.
  - zazen 8y ago
    The first person who made a convnet had an intuition. That is to say, an informal heuristic that gave him reasonable grounds to suppose it might work. This is not "theory". You do not understand what "theory" means in an academic context.