4 ms·
> pretraining on JFT increases the accuracy means that the model is generalizing not necessarily, it may be mostly a bonus of the transfer, especially consider
by trhway 6y ago
> pretraining on JFT increases the accuracy means that the model is generalizing
not necessarily, it may be mostly a bonus of the transfer, especially considering that JFT is that much larger - getting for example the first conv layers kernels to converge to Gabor-like takes time, yet those layers kernels are very similar across the well trained image nets (and there were some works showing that it is optimal in a sense, and that it is one of the reasons it is in our visual cortex) and thus transferrable, and can practically be treated as fixed in the new model (especially if those layers were pretrained on very large model and reached the state of generic feature extraction). I suspect the similar is applicable for the low level feature aggregating layers too.