3 ms·
> There are a few reasonable priors, like initializing and freezing the first convolution to whiten the inputs based upon some statistics from the training data
by elcritch 4y ago
> There are a few reasonable priors, like initializing and freezing the first convolution to whiten the inputs based upon some statistics from the training data. That does a shocking amount of work in stabilizing and speeding up training.
Wow, that sounds nifty! Could you elaborate or point to more resources for that sort of technique?
- tbalsam 4y agoSomeone else linked this in a different tree but I do truly love this blogpost series -- you can see the section "input patch whitening" for more details. https://myrtle.ai/learn/how-to-train-your-resnet-8-bag-of-tricks/ https://myrtle.ai/learn/how-to-train-your-resnet-8-bag-of-tr... Alternatively, you can read through the direct code here if you're willing to wade through some of the mathy code on it: https://github.com/tysam-code/hlb-CIFAR10/blob/main/main.py#L250 https://github.com/tysam-code/hlb-CIFAR10/blob/main/main.py#...
- elcritch 4y agoThanks! Also that code seems pretty straightforward. Later there's this bit: conv_layer.weight.data = (eigenvectors/torch.sqrt(eigenvalues+eps))[-shape[0]:, :, :, :] ## We don't want to train this, since this is implicitly whitening over the whole dataset Oh it looks like you then just "remove" the correlated data.. Actually, you're just pre-"padding" the basic (gaussian?) covariances. Ah so then you'd skip training basic statistical information that's already easy to calculate directly. Clever!