charleshmartin

5 ms·

1.
Detecting Overfit Layers without any Data(twitter.com)
1 points by charleshmartin 8mo ago|1 comments
2.
by charleshmartin 8mo ago
If you train a model for too long, it may overfit it's training data. Not surprising, this has been know for like forever. But did you know you can detect the signatures of overfitting in the layer weight matrices directly, without nee
3.
by charleshmartin 8mo ago
Right. If the dynamics of training are governed by RG flow, then the best optimization path should remove redundant directions, as specified by the RG operator(s)
4.
Setol: SemiEmpirical Theory of (Deep) Learning(arxiv.org)
6 points by charleshmartin 1y ago|3 comments
5.
by charleshmartin 1y ago
We present a SemiEmpirical Theory of Learning (SETOL) that explains the remarkable performance of State-Of-The-Art (SOTA) Neural Networks (NNs). We provide a formal explanation of the origin of the fundamental quantities in the phenomenolog
6.
Setol: A SemiEmpirical of (Deep) Learning(overleaf.com)
4 points by charleshmartin 2y ago|1 comments
7.
by charleshmartin 2y ago
SETOL is a theory of NN layer convergence. It argues that the individual layers of NN converge at different rates, and the 'Ideal' state of convergence can be detected simply by looking at the spectral properties of the layer weig
8.
The Magic of Mistral, a Story of Dragon Kings(calculatedcontent.com)
1 points by charleshmartin 3y ago|0 comments
9.
Evaluating Fine-Tuned LLMs with WeightWatcher(calculatedcontent.com)
2 points by charleshmartin 3y ago|0 comments
10.
by charleshmartin 3y ago
There are good theoretical reasons behind this as well https://calculatedcontent.com/2023/02/01/deep-learning-and-e...
11.
by charleshmartin 4y ago
Envy
12.
by charleshmartin 4y ago
ENVY
13.
Deep Learning and Effective Correlation Spaces(calculatedcontent.com)
1 points by charleshmartin 4y ago|0 comments
14.
Weightwatcher: Data-Free Diagnostics for Deep Learning(weightwatcher.ai)
2 points by charleshmartin 4y ago|1 comments
15.
by charleshmartin 4y ago
WeightWatcher (w|w) is an open-source, diagnostic tool for analyzing Deep Neural Networks (DNN), without needing access to training or even test data. It is based on theoretical research into Why Deep Learning Works and uses the Theory of H
16.
Better than BERT: Pick your best model(calculatedcontent.com)
1 points by charleshmartin 4y ago|0 comments
17.
by charleshmartin 4y ago
https://calculatedcontent.com/2015/03/25/why-does-deep-learn...
18.
by charleshmartin 4y ago
What can they predict with the theory ?
19.
Predicting trends in the quality of state-of-the-art neural networks(nature.com)
1 points by charleshmartin 5y ago|0 comments
20.
by charleshmartin 5y ago
Oh and here's a 2 hour deep dive into the theory https://bluejeans.com/playback/s/cyQnC9EZZSB9HlHDjB1Zpu3fs0J...
21.
by charleshmartin 5y ago
agreed!
22.
by charleshmartin 5y ago
That's fine. Here's a few of my online talks: UC Berkeley / ICSI: https://www.youtube.com/watch?v=6Zgul4oygMc Stanford ICME: https://www.youtube.com/watch?v=PQUItQi-B-I and a couple of Mikes
23.
by charleshmartin 5y ago
Here's an alternative approach, that actually provides real world results https://calculatedcontent.com/2019/12/03/towards-a-new-theor... Using techniques from statistical mechanics and strongly correlat
24.
by charleshmartin 5y ago
Great question. 4 is at the high edge of the fat tailed universality class. Most high performing models have alpha approaching 2, or at least below 3. See Figure 8(a) in the Nature paper, and our upcoming JMLR paper https://ar
25.
How to tell if you have trained your model with enough data(calculatedcontent.com)
2 points by charleshmartin 5y ago|2 comments
26.
Predicting trends in the quality of state-of-the-art neural networks(nature.com)
1 points by charleshmartin 5y ago|0 comments
27.
by charleshmartin 5y ago
You can check for some signatures of over-fitting using the weightwatcher tool https://calculatedcontent.com/2021/04/04/are-your-models-ove... The tool identifies weight matrices that display atypical behavio
28.
Is Your Model Overtrained?(calculatedcontent.com)
2 points by charleshmartin 5y ago|1 comments
29.
by charleshmartin 6y ago
Agreed
30.
by charleshmartin 6y ago
Learning physics is not the same as learning to do physics. If you want to learn how to really do it, you need to work problems. This means you to learn the techniques, and you need problem books, with solved examples, that you can work th

More ›