2 ms·
the square root is from PCA/ZCA whitening, what it does it it makes empirical covariance of gradients become identity, so they become decorellated, which is exa
by big-chungus4 11mo ago
the square root is from PCA/ZCA whitening, what it does it it makes empirical covariance of gradients become identity, so they become decorellated, which is exactly what hessian does on a quadratic objective by the way
- big-chungus4 11mo agohttps://en.wikipedia.org/wiki/Whitening_transformation https://en.wikipedia.org/wiki/Whitening_transformation for ZCA whitening