3 ms·
Is it just me or is anyone else bothered by the lack of theory here? The results are great, but seem so heuristical and ad-hoc, like somebody tried lots of thin
by pron 10y ago
Is it just me or is anyone else bothered by the lack of theory here? The results are great, but seem so heuristical and ad-hoc, like somebody tried lots of things and one seemed to work, but there's no clear idea as to why, and certainly no general theory tying things together.
- visarga 10y agoIt's very experimental. The great thing about them is that they open the possibility of building new types of interfaces like memory, stack, tape, database or search engine that work with neural networks in inference. See https://arxiv.org/pdf/1505.00521v3.pdf https://arxiv.org/pdf/1505.00521v3.pdf
- zodiac 10y agoI can't speak to the RL methods used here, but wrt to the architecture they designed, I think it might seem less ad-hoc if you know that generally speaking new neural net architectures work well when they 1) can do some useful computation 2) with a good loss function 3) with a small number of parameters 4) in a differentiable way 5) where the gradients can flow So for instance LSTMs are better than RNNs due to (5), convnets are good due to (3), etc.
- argonaut 10y agoYou've just numbered 5 different rules of thumb... That hardly supports your idea that things are not adhoc.
- zodiac 10y agoFirstly, (1) and (2) are not rules of thumbs for building nice architectures, but basic properties about supervised learning and what kind of problems it can solved. Even if it were 5 rules of thumb instead of 3, "design an architecture to satisfy 5 rules" is a lot less ad-hoc than "somebody tried lots of things and one seemed to work", which was what I was replying to. We also know why each of these rules are important.
- argonaut 10y agoI'll concede that 1) and 2) are basic properties that an educated layperson would understand after reading a few ML blog posts. Hardly a working theory of neural nets. They're still rules of thumb. You've argued that they are less adhoc than if they didn't exist. That's misses the point: that is still enormously adhoc.
- zardo 10y agoRigour should be a signal to the historian that the maps have been made, and the real explorers have gone elsewhere.
- pron 10y agoI don't think "real mathematicians" do anything less worthy, novel or even revolutionary than "real explorers". Newton's contribution wasn't in collecting data about the motion of the planets or even in discovering patterns in that motion, but in uncovering laws. At this point in time (more precisely, in the past three decades), the study of neural networks feels like ancient astronomy (aside from some early theoretical results); we're in the data-collection and pattern-search phase. The scientific revolution is yet to happen.
- zardo 10y ago>we're in the data-collection and pattern-search phase. The scientific revolution is yet to happen. Sure, and I would call that an age of exploration. I just take issue with criticizing work in a field with no great explanatory theory, for a lack of theory. There are only two routes a researcher can take to address the criticism, abandon work in the field, or come up with the missing breakthrough theory. It's sort of like saying no one should have bothered with biology until after Darwin figured out what was going on.
- pron 10y agoYou're right. I didn't mean it as criticism, although it may have come out this way, only I decided to read a modern NN paper after not having touched NNs for almost 20 years, and was surprised to learn that there's little if any new theory.
- deleted 10y ago[deleted]
- pron 10y agoSure, but that's engineering (inspired by what theory of basic NN we do have), not theory.
- Houshalter 10y agoIt's not really ad hoc. Neural nets are well studied, but they have very limited memory. They attempted to solve this problem by giving a neural net an unlimited external tape memory, inspired by Turing machines. In order to train a neural net with backpropagation, every step has to be completely continuous and differentiable. So they made a continuous valued tape.
- argonaut 10y ago> inspired by Turing machines This is the very spirit of ad-hoc engineering and trial and error: trying some architecture out because it resembles / has high-level connections to some other unrelated concept.
- zodiac 10y agoYou deliberately ignored the other part of his explanation. They didn't try out using an external tape just because they were inspired by Turing machines, but because previous RNN neural net architectures had a problem in that memory could only be stored in the hidden states, which is expensive. "Finding a limitation in existing architectures and designing a new architecture that avoids those limitations" is not ad-hoc. Additionally, the concept of a Turing machine is hardly "unrelated" to the goal of creating architectures that can learn efficiently, since the registers + external memory architecture of computers has proven to be useful for cheaply and quickly doing real-life tasks using short programs, something we would of course want our trained ML models to do.
- pron 10y agoThis isn't shooting completely in the dark, and obviously there's clear motivation, but motivation, inspiration and engineering are still not theory. Let me put it another way: that paper reads like a technical report or a paper in an experimental science. It does not all resemble a "normal" academic CS or math paper. That's fine, maybe that's the stage this discipline is at, but I was surprised to see that there's still no new theory since when I worked with NNs in the 90s.
- argonaut 10y agoDeep learning is very adhoc and experimental right now. Anyone reading through the literature would see that. That's why I'm pessimistic about AGI, but very optimistic about short-mid-term industry applications.