6 ms·
I'm sorry but this comment is just at odds with the current literature in deep learning. A few very prominent researchers have been trying to use information th
by mathinpens 8y ago
I'm sorry but this comment is just at odds with the current literature in deep learning.
A few very prominent researchers have been trying to use information theoretic concepts to explain generalization for a while now-see
the somewhat controversial branch Naftali Tishby developed...However many other papers apply 'actual information theory'....
https://arxiv.org/search/?query=deep+learning+information+theory&searchtype=all&order=&size=50 https://arxiv.org/search/?query=deep+learning+information+th...
- derefr 8y agoI think the GP poster meant to contrast this paper not with other papers, but with the junk explanations for "how deep learning works" that science journalism comes up with.
- mathinpens 8y agotrust me some of the junk explanations for how deep learning works originate from researchers. we really just don't have a convincing picture about why these networks work or don't work in a theoretical way rather than empirical/heuristic handwaves
- crb002 8y agoBollocks see my comment. We know worst case bounds for Kolmogorov random transformations of a finite set, the most complex learning task there is.
- mathinpens 8y agohuh? not even wrong......
- sp332 8y agoGood. I hope to see more of them posted to HN.
- mathinpens 8y agoif you are interested in this type of analysis are you familiar with the tishby papers? you will at the very least find them enjoyable/mind opening... but some of his key claims are probably contradicted by empirical results. the original paper: https://arxiv.org/abs/1503.02406 https://arxiv.org/abs/1503.02406 a refutation:(with very harsh/unprofessional words by tishby in counter-refutation) https://openreview.net/forum?id=ry_WPG-A-¬eId=ry_WPG-A- https://openreview.net/forum?id=ry_WPG-A-¬eId=ry_WPG-A-
- wangii 8y agoI found his talk on youtube, remarkable: https://www.youtube.com/watch?v=XL07WEc2TRI https://www.youtube.com/watch?v=XL07WEc2TRI
- mathinpens 8y agoHe's a good speaker/teacher but I think there are serious problems with his work- mainly that a lot of his results collapse when tested on more general architectures.