5 ms·
Man, it's good to see some actual information theory applied to deep learning for a change.
by sp332 9y ago
Man, it's good to see some actual information theory applied to deep learning for a change.
- mathinpens 9y agoI'm sorry but this comment is just at odds with the current literature in deep learning. A few very prominent researchers have been trying to use information theoretic concepts to explain generalization for a while now-see the somewhat controversial branch Naftali Tishby developed...However many other papers apply 'actual information theory'.... https://arxiv.org/search/?query=deep+learning+information+theory&searchtype=all&order=&size=50 https://arxiv.org/search/?query=deep+learning+information+th...
- derefr 9y agoI think the GP poster meant to contrast this paper not with other papers, but with the junk explanations for "how deep learning works" that science journalism comes up with.
- mathinpens 9y agotrust me some of the junk explanations for how deep learning works originate from researchers. we really just don't have a convincing picture about why these networks work or don't work in a theoretical way rather than empirical/heuristic handwaves
- crb002 9y agoBollocks see my comment. We know worst case bounds for Kolmogorov random transformations of a finite set, the most complex learning task there is.
- mathinpens 9y agohuh? not even wrong......
- sp332 9y agoGood. I hope to see more of them posted to HN.
- mathinpens 9y agoif you are interested in this type of analysis are you familiar with the tishby papers? you will at the very least find them enjoyable/mind opening... but some of his key claims are probably contradicted by empirical results. the original paper: https://arxiv.org/abs/1503.02406 https://arxiv.org/abs/1503.02406 a refutation:(with very harsh/unprofessional words by tishby in counter-refutation) https://openreview.net/forum?id=ry_WPG-A-¬eId=ry_WPG-A- https://openreview.net/forum?id=ry_WPG-A-¬eId=ry_WPG-A-
- wangii 9y agoI found his talk on youtube, remarkable: https://www.youtube.com/watch?v=XL07WEc2TRI https://www.youtube.com/watch?v=XL07WEc2TRI
- mathinpens 9y agoHe's a good speaker/teacher but I think there are serious problems with his work- mainly that a lot of his results collapse when tested on more general architectures.
- Cacti 9y agoThere's papers posted to arXiv every day using information theory to analyze deep learning...
- deleted 9y ago[deleted]