10 ms·
I can't speak to the RL methods used here, but wrt to the architecture they designed, I think it might seem less ad-hoc if you know that generally speaking new
by zodiac 10y ago
I can't speak to the RL methods used here, but wrt to the architecture they designed, I think it might seem less ad-hoc if you know that generally speaking new neural net architectures work well when they
1) can do some useful computation
2) with a good loss function
3) with a small number of parameters
4) in a differentiable way
5) where the gradients can flow
So for instance LSTMs are better than RNNs due to (5), convnets are good due to (3), etc.
- argonaut 10y agoYou've just numbered 5 different rules of thumb... That hardly supports your idea that things are not adhoc.
- zodiac 10y agoFirstly, (1) and (2) are not rules of thumbs for building nice architectures, but basic properties about supervised learning and what kind of problems it can solved. Even if it were 5 rules of thumb instead of 3, "design an architecture to satisfy 5 rules" is a lot less ad-hoc than "somebody tried lots of things and one seemed to work", which was what I was replying to. We also know why each of these rules are important.
- argonaut 10y agoI'll concede that 1) and 2) are basic properties that an educated layperson would understand after reading a few ML blog posts. Hardly a working theory of neural nets. They're still rules of thumb. You've argued that they are less adhoc than if they didn't exist. That's misses the point: that is still enormously adhoc.
- zardo 10y agoRigour should be a signal to the historian that the maps have been made, and the real explorers have gone elsewhere.
- pron 10y agoI don't think "real mathematicians" do anything less worthy, novel or even revolutionary than "real explorers". Newton's contribution wasn't in collecting data about the motion of the planets or even in discovering patterns in that motion, but in uncovering laws. At this point in time (more precisely, in the past three decades), the study of neural networks feels like ancient astronomy (aside from some early theoretical results); we're in the data-collection and pattern-search phase. The scientific revolution is yet to happen.
- zardo 10y ago>we're in the data-collection and pattern-search phase. The scientific revolution is yet to happen. Sure, and I would call that an age of exploration. I just take issue with criticizing work in a field with no great explanatory theory, for a lack of theory. There are only two routes a researcher can take to address the criticism, abandon work in the field, or come up with the missing breakthrough theory. It's sort of like saying no one should have bothered with biology until after Darwin figured out what was going on.
- pron 10y agoYou're right. I didn't mean it as criticism, although it may have come out this way, only I decided to read a modern NN paper after not having touched NNs for almost 20 years, and was surprised to learn that there's little if any new theory.
- deleted 10y ago[deleted]
- pron 10y agoSure, but that's engineering (inspired by what theory of basic NN we do have), not theory.