4 ms·
You deliberately ignored the other part of his explanation. They didn't try out using an external tape just because they were inspired by Turing machines, but b
by zodiac 10y ago
You deliberately ignored the other part of his explanation. They didn't try out using an external tape just because they were inspired by Turing machines, but because previous RNN neural net architectures had a problem in that memory could only be stored in the hidden states, which is expensive. "Finding a limitation in existing architectures and designing a new architecture that avoids those limitations" is not ad-hoc.
Additionally, the concept of a Turing machine is hardly "unrelated" to the goal of creating architectures that can learn efficiently, since the registers + external memory architecture of computers has proven to be useful for cheaply and quickly doing real-life tasks using short programs, something we would of course want our trained ML models to do.
- pron 10y agoThis isn't shooting completely in the dark, and obviously there's clear motivation, but motivation, inspiration and engineering are still not theory. Let me put it another way: that paper reads like a technical report or a paper in an experimental science. It does not all resemble a "normal" academic CS or math paper. That's fine, maybe that's the stage this discipline is at, but I was surprised to see that there's still no new theory since when I worked with NNs in the 90s.
- zodiac 10y agoI agree with this. I think it will be a while before we get a decent theory, though, and it's not because no one is trying, but it seems too hard to me. For instance, the way I see it, we don't even know very well how the low-dimensional space of (for example) "naturally occurring pictures" is embedded into the high-dimensional space of "all possible pictures". So we can't even precisely state what we want our NNs to do, much less why they work well. Put another way, if we ask a mathematician "so why is it that when I use gradient descent on this architecture, I perform reasonably well on unseen examples?", she would reply, "ok, what distribution did you draw your training and unseen examples from?", and we can't answer that.
- argonaut 10y agoCompletely agreed with pron's comment. Also, the process by which they designed the new architecture is adhoc. They saw a problem, and then tried a bunch of solutions, and got something to work. Sure, their effort was guided by intuition, but that is adhoc. My point is, there was no theory they worked out saying "you should do this," or "these are the non-empirical reasons this method is superior." I'm not negative on neural nets. I actually like that the field is relatively shallow and fast-moving. I just dislike it when people on HN valorize neural nets to be more than they are.
- zodiac 10y agoYou can't just say that things are either adhoc or not adhoc - it's a spectrum. On one end we will have engineers coming up with architectures by consulting a RNG and then testing out those architectures. On the other end we will have a theory, the theory will make 100% good predictions about which designs will perform well, and we can just send send those designs to a factory to fabricate into asics, no testing needed. The adhoc-ness of how we design neural net architectures, CPUs, GPUs, high-temperature suerconductors, etc all fall on this spectrum, just at different places. pron's original comment made it sound like designing NNs were on the extreme ad-hoc end of the spectrum, I was just refuting that.