3 ms·
Then why not start with a model that has all the weights zeroes out and start absorbing different models?
by m3kw9 3y ago
Then why not start with a model that has all the weights zeroes out and start absorbing different models?
- l33tman 3y agoIt would be cool if there was like a threshold in training from scratch, where after this actually works for adding higher level knowledge. So you would start training it like usual to get it to absorb generic langauge skills and reasoning but save all the domain knowledge absorption for merging later
- skybrian 3y agoThe weights need to be connected to something interesting. Pre-training is how they get them all connected up, and fine-tuning is how they find which weights it would be useful to change.
- stoniejohnson 3y agoPretraining determines the weights (e.g. the connections), fine-tuning let's you change some subset of the weights (e.g. the final layers) with a smaller chunk of task-specific data.
- yumraj 3y agoIsn’t x + 0 = x? So what’s the benefit of starting with 0 when you can just start with x I know it’s over simplistic above, but serious question, please ELI5..
- m3kw9 3y agoAnd then you apply a different models weights using the papers methods and then another