5 ms·
DL researcher here, it's also hard I think because experimentally many of us have noted (also some research on this) that there's a critical early phase to lear
by tysam_and 3y ago
DL researcher here, it's also hard I think because experimentally many of us have noted (also some research on this) that there's a critical early phase to learning that conditions the network a certain way, and adding and/or removing layers later seems to actually be quite difficult (removing easier than adding, by far, esp w/ something like variational dropout [yes, i cite the old deep magicks]: https://arxiv.org/abs/1701.05369 https://arxiv.org/abs/1701.05369)
- hiddencost 3y agoI had success doubling the width of a layer by copying, adding epsilon noise and then fine tuning. Transform from [X] to [X X+epsilon]. In that case I was trying to fuse two signals that had fairly similar characteristics.
- tysam_and 3y agoYes, unfortunately I have yet to beat that technique in wallclock convergence speed vs just using the larger network from the start. :'(((( Whoever figures out how to clearly and effectively do it consistently faster than a 'full network from the start' version will open up an entirely new subfield of neural network training efficiency. :'))))