4 ms·
tl;dr ordinary gradient descent is sensitive to the parametrization of the problem but natural gradient is not. This is an important fact (and one that is fairl
by simonster 8y ago
tl;dr ordinary gradient descent is sensitive to the parametrization of the problem but natural gradient is not. This is an important fact (and one that is fairly well-known within the ML community), but it is not totally clear to me why it should be particularly relevant to neuroscience.
- bjourne 8y agoIsn't it only the trajectory that is sensitive to the parametrization? Afaik, the location of the minimas do not depend on it.
- simonster 8y agoYep, the minima are in the same locations. However, if the problem has multiple minima, then the parametrization can affect which minimum gradient descent actually reaches.