4 ms·

Hello Smerity, I have read the deeplearning book and I think it's great that you know Yarin. His PhD thesis is the closest thing to a survey paper on uncertain

Hello Smerity, I have read the deeplearning book and I think it's great that you know Yarin. His PhD thesis is the closest thing to a survey paper on uncertainty in Deep Learning. I also think that while the original dropout paper is nice, Yarin's paper extends this method in a very useful manner. Now, let me address your other points: 1. This is not for people who want to get started in deep learning. Or it could be. Do you know how I got started into mathematics? 2. The dubious neuroscience connection isn't dubious at all. In particular, I would also advise that you read 'Towards an integration of deep learning and neuroscience' by Marblestone et al. If you still disagree, I could assure you that Dr Kriegeskorte, Konrad Kording, Professor Bengio and many others are of a different opinion. Furthermore, the language of statistical learning has shown its limitations. Consider Srivastava's paper on Locally Competitive Networks and every other paper on deep rectifier networks for example. As the complexity of deep models approach those of biological systems, I have no doubt we'll be using similar tools to understand these models. fyi: I'm currently working on deep rectifier networks. Aidan

Smerity 9y ago
Hey Aidan, As noted, it's not a criticism of a reading list tailored for you, I simply feel it's a really confusing reading list given the likely audience that will arrive there from Hacker News and the title. I also don't think it can be pieced together to form a good introductory reading list or that the contents as stated are "foundational" yet given the progression and holes. Your (1) confuses me still - I admittedly don't know how you got started in mathematics or how that's relevant here? Regarding (2), there certainly is a place for neuroscience to influencing thinking or introduce new ideas, and I'm certainly not disregarding the entirety of that potential intersection, but it's a very specific field that is still largely disconnected from the practical application of neural networks. This may change, slightly or substantially, in the future, but thinking again with the lens of the audience of Hacker News (where the incorrect adages of the style "deep learning learns just like a human brain" gets thrown around frequently), I rarely want to exaggerate the influence that neuroscience has on the field at this stage. I'd also lightly note that appealing to authority isn't an argument, though do note the author's pedigree.
- aidanrocke 9y ago
  Good morning Smerity, I'm sorry if my response sounded harsh. It wasn't my goal to confuse readers and I admit that it's a work in progress. That said I must clarify a few things as I realise that I left out important details: 1. Regarding neuroscience: Those researchers I've mentioned(Kording et al.) are actually people I've discussed this issue with recently. That's the context. However, I must say that I thoroughly agree with you that "deep learning learns just like a human brain" is a very inaccurate statement and a reflection of the imprecise understanding many deep learning practitioners(and some researchers) have of their own field. 2. My approach to math: Was certainly not the approach used by deep learning tutorials. The issue I have with the proliferation of these tutorials on the internet is that they lead to familiarity rather than understanding. How to reach such a theoretical understanding is debatable and I'd like to encourage a debate. In fact, this list is part of a bigger project which I'd like to collaborate on with others in the near future. You can think of it as the early stages of a Bourbaki project for deep learning. :) Who was Bourbaki: http://www.ams.org/notices/200709/tx070901150p.pdf http://www.ams.org/notices/200709/tx070901150p.pdf

Smerity 9y ago

Hey Aidan, As noted, it's not a criticism of a reading list tailored for you, I simply feel it's a really confusing reading list given the likely audience that will arrive there from Hacker News and the title. I also don't think it can be pieced together to form a good introductory reading list or that the contents as stated are "foundational" yet given the progression and holes. Your (1) confuses me still - I admittedly don't know how you got started in mathematics or how that's relevant here? Regarding (2), there certainly is a place for neuroscience to influencing thinking or introduce new ideas, and I'm certainly not disregarding the entirety of that potential intersection, but it's a very specific field that is still largely disconnected from the practical application of neural networks. This may change, slightly or substantially, in the future, but thinking again with the lens of the audience of Hacker News (where the incorrect adages of the style "deep learning learns just like a human brain" gets thrown around frequently), I rarely want to exaggerate the influence that neuroscience has on the field at this stage. I'd also lightly note that appealing to authority isn't an argument, though do note the author's pedigree.

aidanrocke 9y ago
Good morning Smerity, I'm sorry if my response sounded harsh. It wasn't my goal to confuse readers and I admit that it's a work in progress. That said I must clarify a few things as I realise that I left out important details: 1. Regarding neuroscience: Those researchers I've mentioned(Kording et al.) are actually people I've discussed this issue with recently. That's the context. However, I must say that I thoroughly agree with you that "deep learning learns just like a human brain" is a very inaccurate statement and a reflection of the imprecise understanding many deep learning practitioners(and some researchers) have of their own field. 2. My approach to math: Was certainly not the approach used by deep learning tutorials. The issue I have with the proliferation of these tutorials on the internet is that they lead to familiarity rather than understanding. How to reach such a theoretical understanding is debatable and I'd like to encourage a debate. In fact, this list is part of a bigger project which I'd like to collaborate on with others in the near future. You can think of it as the early stages of a Bourbaki project for deep learning. :) Who was Bourbaki: http://www.ams.org/notices/200709/tx070901150p.pdf http://www.ams.org/notices/200709/tx070901150p.pdf