3 ms·
It's not just about the sum. The upper and lower indices are semantically crucial. Eg: vectors have upper indices and gradients have lower indices. It's nonsens
by ssivark 8y ago
It's not just about the sum. The upper and lower indices are semantically crucial. Eg: vectors have upper indices and gradients have lower indices. It's nonsensical to add the two, so the core step in gradient descent doesn't make sense. To add the two, you need the metric (Hessian) to raise the index of the gradient. And just like that, voila, you've discovered natural gradients!
- mmmmpancakes 8y agoGradients are vectors. You can certainly add them to other vectors. I think what you mean to say is that you can not add vectors and covectors. The differential of a function is a 1-form. Pointwise, it is a covector. This data is the natural starting point from which to compute gradients. Metrics can be used to convert covectors into vectors and forms into vector fields. The result is the thing you call the natural gradient. If you do this with the standard metric on R^n, then you get the vector field which you call the gradient.