3 ms·
Someone else linked this in a different tree but I do truly love this blogpost series -- you can see the section "input patch whitening" for more details. https
by tbalsam 4y ago
Someone else linked this in a different tree but I do truly love this blogpost series -- you can see the section "input patch whitening" for more details. https://myrtle.ai/learn/how-to-train-your-resnet-8-bag-of-tricks/ https://myrtle.ai/learn/how-to-train-your-resnet-8-bag-of-tr...
Alternatively, you can read through the direct code here if you're willing to wade through some of the mathy code on it: https://github.com/tysam-code/hlb-CIFAR10/blob/main/main.py#L250 https://github.com/tysam-code/hlb-CIFAR10/blob/main/main.py#...
- elcritch 4y agoThanks! Also that code seems pretty straightforward. Later there's this bit: conv_layer.weight.data = (eigenvectors/torch.sqrt(eigenvalues+eps))[-shape[0]:, :, :, :] ## We don't want to train this, since this is implicitly whitening over the whole dataset Oh it looks like you then just "remove" the correlated data.. Actually, you're just pre-"padding" the basic (gaussian?) covariances. Ah so then you'd skip training basic statistical information that's already easy to calculate directly. Clever!