3 ms·

As a transplant to the ML world who migrated from physics I do find it funny that people in ML will insist that gradients are vectors. Once you get to general

As a transplant to the ML world who migrated from physics I do find it funny that people in ML will insist that gradients are vectors. Once you get to general relativity your physics professors will beat into you the fact that gradients are not vectors. (They are one-forms.) But in the ML context it's a distinction without a difference, unless you're in a really specialized corner of the field like information geometry.

ReleaseCandidat 3y ago
> Once you get to general relativity your physics professors will beat into you the fact that gradients are not vectors. In linear algebra you should learn this as soon as you hear about it. Is the english Wikipedia page correct in that you always use nabla for the gradient? https://en.wikipedia.org/wiki/Gradient#Generalizations https://en.wikipedia.org/wiki/Gradient#Generalizations Because `grad f = ∇f` holds in scalar fields f only (not in vector or tensor fields (tensors that aren't scalars)).
- antognini 3y ago
  Even if you are taking the gradient of a scalar field, the resulting object is not a vector. The difference is that if you apply some transformation to your coordinate system a gradient transforms differently from a normal vector. In the ML context it generally doesn't matter because you don't do these transformations (and more generally there is no metric for your space). So you can just treat a gradient as an array of numbers. But in a physical context the distinction starts to matter, at least on a manifold with curvature. The one place it does matter in ML (that I'm aware of) is information geometry, where you try to do a transformation from the normal coordinate space to a more "natural" coordinate system that is based on the Fisher information, and this introduces curvature into model.

3 ms·

As a transplant to the ML world who migrated from physics I do find it funny that people in ML will insist that gradients are vectors. Once you get to general

by antognini 3y ago

ReleaseCandidat 3y ago
> Once you get to general relativity your physics professors will beat into you the fact that gradients are not vectors. In linear algebra you should learn this as soon as you hear about it. Is the english Wikipedia page correct in that you always use nabla for the gradient? https://en.wikipedia.org/wiki/Gradient#Generalizations https://en.wikipedia.org/wiki/Gradient#Generalizations Because `grad f = ∇f` holds in scalar fields f only (not in vector or tensor fields (tensors that aren't scalars)).
- antognini 3y ago
  Even if you are taking the gradient of a scalar field, the resulting object is not a vector. The difference is that if you apply some transformation to your coordinate system a gradient transforms differently from a normal vector. In the ML context it generally doesn't matter because you don't do these transformations (and more generally there is no metric for your space). So you can just treat a gradient as an array of numbers. But in a physical context the distinction starts to matter, at least on a manifold with curvature. The one place it does matter in ML (that I'm aware of) is information geometry, where you try to do a transformation from the normal coordinate space to a more "natural" coordinate system that is based on the Fisher information, and this introduces curvature into model.