3 ms·
Here's an alternative approach, that actually provides real world results https://calculatedcontent.com/2019/12/03/towards-a-new-theory-of-learning-statistical
by charleshmartin 5y ago
Here's an alternative approach, that actually provides real world results
https://calculatedcontent.com/2019/12/03/towards-a-new-theory-of-learning-statistical-mechanics-of-deep-neural-networks/ https://calculatedcontent.com/2019/12/03/towards-a-new-theor...
Using techniques from statistical mechanics and strongly correlated systems, we can compute the average-case-behavior of a real world DNN
We believe we can reproduce some of the results of the NTK by using a Gaussian Random Matrix. But if we use a more realistic, heavy tailed matrix, we get more practical results
A early fork of the theory has been published in JMLR
https://arxiv.org/abs/1810.01075 https://arxiv.org/abs/1810.01075
and the empirical results in Nature Communications
https://www.nature.com/articles/s41467-021-24025-8 https://www.nature.com/articles/s41467-021-24025-8
and we have an open source tool , weightwatcher, which can be used in production
pip install weightwatcher
https://github.com/CalculatedContent/WeightWatcher https://github.com/CalculatedContent/WeightWatcher
Please give it a try and let me know if it is useful to you
- thisnormaluser 5y agoNature Communications is not Nature. But whatever.
- chaxor 5y agoI actually have a preference towards Nature Communications over Nature, as Nature Communications actually allows more pages for explanation and as such typically has more rigorous research. On the flip side, Nature often has bolder claims with less convincing evidence, which often reads similar to pop-sci articles.
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- charleshmartin 5y agoThat's fine. Here's a few of my online talks: UC Berkeley / ICSI: https://www.youtube.com/watch?v=6Zgul4oygMc https://www.youtube.com/watch?v=6Zgul4oygMc Stanford ICME: https://www.youtube.com/watch?v=PQUItQi-B-I https://www.youtube.com/watch?v=PQUItQi-B-I and a couple of Mikes's Institute for Pure & Applied Mathematics (IPAM) : https://www.youtube.com/watch?v=fmVuNRKsQa8 https://www.youtube.com/watch?v=fmVuNRKsQa8 Physics Informed Machine Learning: https://www.youtube.com/watch?v=eXhwLtjtUsI https://www.youtube.com/watch?v=eXhwLtjtUsI and our KDD workshop KDD: https://dl.acm.org/doi/abs/10.1145/3292500.3332294 https://dl.acm.org/doi/abs/10.1145/3292500.3332294 many more talks and papers available And my favorite, a podcast that has featured LeCun himself: https://blog.rebellionresearch.com/blog/theoretical-physicist-dr-charles-martin-on-deep-learning https://blog.rebellionresearch.com/blog/theoretical-physicis...
- charleshmartin 5y agoOh and here's a 2 hour deep dive into the theory https://bluejeans.com/playback/s/cyQnC9EZZSB9HlHDjB1Zpu3fs0J71i4kPqN3ewCzbLm5AxKQzJ12gqwteONlsEtM https://bluejeans.com/playback/s/cyQnC9EZZSB9HlHDjB1Zpu3fs0J...