4 ms·
Really? Can you honestly say with a straight face that there is no theory behind why convnets are the right choice? Or no theory behind how an LSTM is structu
by dnautics 8y ago
Really? Can you honestly say with a straight face that there is no theory behind why convnets are the right choice? Or no theory behind how an LSTM is structured?
- zazen 8y agoThere is plenty of room to be dissatisfied with the current state of deep learning theory without making absolute claims that there is "no theory" whatsoever, which GP clearly isn't. If you wish to challenge GP's comment, you could do so constructively by discussing any specific theoretical contributions you feel GP may have overlooked.
- dnautics 8y ago> You can make a simple framework of 3-4 neurons and observe how it works. Then you make it bigger and it works differently. No one properly explained why this happens.
- minkzilla 8y agoCan you cite an explanation? I'd love to read one.
- dnautics 8y agoSure. you can watch one that's in very easy to understand plain english. prereq: https://www.youtube.com/watch?v=C_zFhWdM4ic https://www.youtube.com/watch?v=C_zFhWdM4ic substance: https://www.youtube.com/watch?v=py5byOOHZM8 https://www.youtube.com/watch?v=py5byOOHZM8 I mean what is the difference between like using 50 layers versus 75 layers? Ok, yeah we don't know, but to presume that we don't understand anything beyond 3-4 neurons is just nonsense. By comparison. Go look at any synthetic chemistry paper. There's going to be crazy procedures. One time Phil Baran used Xenon Tetrafluoride as an oxidizing agent. And yeah, for some reason none of the other standard oxidizing agents worked. We don't know why. Hell it could have been that the grad student had bad hands with one of the standard ones. We can make a SWAG an to why XeF4 worked. But we'd be fooling ourselves to claim "we know", so we don't say we do. Does that discredit synthetic chemistry as an enterprise? Does that mean synthetic chemistry is not backed by theory?
- zazen 8y agoThat video is an introduction to the concept of CNNs for beginners. It explains in informal terms the structure and mechanics of a CNN, and probably gets around to discussing why it might work in very broad terms. This is extremely far from what academically inclined people are talking about when they are using the word "theory". People are hoping to see, ultimately, mathematical proofs that neural networks can e.g. efficiently learn target functions given a certain amount of training data and a certain number of trainable parameters. To address your chemistry analogy: no, it doesn't discredit synthetic chemistry as an enterprise, it just means there is a gap in the theory. Similarly, the people bemoaning gaps in the deep learning theory aren't trying to discredit deep learning as an enterprise: the practical successes are very real; the theory is just lagging, as it often has in application-led fields.
- dnautics 8y ago> It explains in informal terms the structure and mechanics of a CNN It actually gets quite formal. you could implement a (non-performant) CNN from everything in that video. The point is, that someone came up with a hypothesis "hey imposing this structure should yield faster convergence to usable results" is totally inductive science, and that is able to be explained quite well and simply from simple primitives that you don't need a math degree to understand really indicates some level of maturity in the field. >People are hoping to see, ultimately, mathematical proofs that neural networks can e.g. efficiently learn target functions given a certain amount of training data and a certain number of trainable parameters. yeah I don't dispute that, but to characterize our understanding of machine learning as not going beyond 3 or 4 neurons is just plain crazy. Moreover, for some RNNs, e.g. character-based LSTMs trained on language models, we can extract feature details. I recall one case (sorry not bothered to find it) where there was a memory line dedicated to detetcting opening and closing quotation marks. Now this is a feature that one would expect to be obvious, or at least more so than other language features, but it's still a rather high level (and successful) attempt to understand how these machines function.
- gambler 8y ago
- gambler 8y ago>Can you honestly say with a straight face that there is no theory behind why convnets are the right choice? There is theory of convnets, but it's more of a post-factum analysis, really, and I'm not sure it's behind much of anything at this point. Researchers use convnets because they work in ways that work for them. Theory: 109 citations. https://www.semanticscholar.org/paper/A-Mathematical-Theory-of-Deep-Convolutional-Neural-Wiatowski-Bx00F6lcskei/4043604116f90200c3b7cc7573a133349a24bb78?navId=citing-papers https://www.semanticscholar.org/paper/A-Mathematical-Theory-... Practice: 42,551 citations. https://www.semanticscholar.org/paper/ImageNet-Classification-with-Deep-Convolutional-Krizhevsky-Sutskever/2315fc6c2c0c4abd2443e26a26e7bb86df8e24cc?navId=citing-papers https://www.semanticscholar.org/paper/ImageNet-Classificatio...
- dnautics 8y ago> Researchers use convnets because they work in ways that work for them. What about the first person who made a convnet? This is basically like saying there is no understanding of the wheel since there are relatively few citations on the theory of the wheel relative to papers that use wheels.
- zazen 8y agoThe first person who made a convnet had an intuition. That is to say, an informal heuristic that gave him reasonable grounds to suppose it might work. This is not "theory". You do not understand what "theory" means in an academic context.