3 ms·
I would like to know if this approach is actually transferring knowledge, or if random initialization is just not the optimal starting place - for example, mayb
by londons_explore 3y ago
I would like to know if this approach is actually transferring knowledge, or if random initialization is just not the optimal starting place - for example, maybe having a different distribution of weights is better.
A good test would be to apply this technique, but then train on a totally different kind of data - for example take a text LLM, apply this technique, but then train on audio/music data and see if this technique reduces training time over random initialization.
- peterfirefly 3y agoSounds like a good test. We need something like this for "non-neural" things like transformers and normalization layers: https://proceedings.mlr.press/v157/skorski21a/skorski21a.pdf https://proceedings.mlr.press/v157/skorski21a/skorski21a.pdf
- ausbah 3y agoa better distribution of weights would be knowledge transfer right? by having something like a stronger prior distribution? also pretty sure this would work because I know some text-image generators has the image network priority trained then frozen during e2e training
- peterfirefly 3y agoNot necessarily. It turns out to really matter how the weights are initialized (randomly, but from which distribution?) for ordinary feed-forward neural nets that only uses standard neurons -- different activation functions have different optimal initialization distributions. https://machinelearningmastery.com/weight-initialization-for-deep-learning-neural-networks/ https://machinelearningmastery.com/weight-initialization-for... A big breakthrough was Kaiming He's work on initialization for ReLU: https://arxiv.org/abs/1502.01852 https://arxiv.org/abs/1502.01852 (Kaiming is the personal name, He is the family name. He Kaiming is the native Chinese order. Lots of citations use "Kaiming" by mistake.)