5 ms·

> Yeah. I don't know why LA courses focus so much on eigenvalues, and don't talk much about SVD. It makes sense to talk about the eigenvalues of an operator on

> Yeah. I don't know why LA courses focus so much on eigenvalues, and don't talk much about SVD. It makes sense to talk about the eigenvalues of an operator on any vector space. But in order for singular values to make sense the vector space needs to have an inner product. So eigenvalues are more general because they make sense even when you haven't chosen an inner product on your vector space. From an HN perspective this kind of consideration is relevant when working on a data set in which there are some arbitrary units. You don't want your model predicting house prices to work differently depending on whether you choose to measure floor size in m^2 or ft^2.

ssivark 6y ago
Could you elaborate on why SVD requires an inner product structure (choice of scale) while eigenvalues do not? Do you just mean that the input and output vector spaces might generically have different units, and so one would need to use some sensible "normalized" units?
- mgraczyk 6y ago
  Singular values of a transform are defined using the "operator norm" of the transform. The norm is defined in terms of the inner product. On the other hand, eigenvalues are defined with using only the transform and scalar multiplication, which does not require an inner product.
  - tsbinz 6y ago
    I'm not sure you need to do that (define singular values via the operator norm), but one thing I don't see a solution for is getting rid of the requirement of the inner product for the definition of the adjoint.
- kkylin 6y ago
  There's probably more than one perspective on this. Here's one: a standard way to approach SVD is to start with a matrix A and consider A^TA and AA^T. Since the latter is symmetric (or hermitian if on a complex vector space, where transpose is replaced by adjoint), by the spectral theorem it has real eigenvalues and orthonormal eigenvectors, from which the SVD can be deduced. But the definition of A^T or A^* depends on a particular choice of basis, and is not a coordinate-invariant concept. If one had an inner product <,>, the adjoint can be defined in a coordinate-independent way using that inner product: A^* is the operator defined by <Au,v> = <u,A^v> for all vectors u and v. (One can check that A^ is uniquely defined.) For a different choice of inner product, one gets a different A^. Note that the operator A^TA comes up naturally when one considers what the action of A does to the length of vectors: |u|^2 = <u,u> |Au|^2 = <Au,Au> = <u,A^Au> Geometrically, the SVD tells us how a linear transformation dilates or contracts space in different directions, e.g., think about what a linear transformation does to a sphere. But all these concepts -- length of vectors, spheres, ellipsoids (images of spheres under linear transformations) -- depend on a choice of inner product.
- jacobolus 6y ago
  The SVD breaks a linear transformation into 3 parts: (1) A rotation in the source space, (2) an axis-aligned non-uniform scaling (if necessary adding/removing dimensions to match the destination space), and (3) another rotation in the destination space. Rotation is only a meaningful concept when you have a metrical structure (think of the full geometry of Euclid’s Elements including circles, perpendicularity, lengths, angles), so if you don’t have a metrical structure the SVD is likewise not really meaningful. Often spaces we deal with using linear algebra only have an affine structure (parallelism is well defined, and lengths can be compared when they are along parallel lines), but do not have a metrical structure (so there is no meaningful way to compare lengths pointed in different directions or measure angles). For example: there is no meaningful concept of the angle between the directions of 5 miles/gallon and 10 miles/gallon, and if you arbitrarily defined one, it would change when you switched from gallons to milliliters or from miles to meters. In practice people still often arbitrarily impose a metrical structure, sometimes based on some heuristic analysis of the data involved, and then use tools like the SVD based on that.
- contravariant 6y ago
  The really short answer is that (one off) the definition(s) of SVDs uses the transpose, which requires a inner product.

ssivark 6y ago

Could you elaborate on why SVD requires an inner product structure (choice of scale) while eigenvalues do not? Do you just mean that the input and output vector spaces might generically have different units, and so one would need to use some sensible "normalized" units?

mgraczyk 6y ago
Singular values of a transform are defined using the "operator norm" of the transform. The norm is defined in terms of the inner product. On the other hand, eigenvalues are defined with using only the transform and scalar multiplication, which does not require an inner product.
- tsbinz 6y ago
  I'm not sure you need to do that (define singular values via the operator norm), but one thing I don't see a solution for is getting rid of the requirement of the inner product for the definition of the adjoint.
kkylin 6y ago
There's probably more than one perspective on this. Here's one: a standard way to approach SVD is to start with a matrix A and consider A^TA and AA^T. Since the latter is symmetric (or hermitian if on a complex vector space, where transpose is replaced by adjoint), by the spectral theorem it has real eigenvalues and orthonormal eigenvectors, from which the SVD can be deduced. But the definition of A^T or A^* depends on a particular choice of basis, and is not a coordinate-invariant concept. If one had an inner product <,>, the adjoint can be defined in a coordinate-independent way using that inner product: A^* is the operator defined by <Au,v> = <u,A^v> for all vectors u and v. (One can check that A^ is uniquely defined.) For a different choice of inner product, one gets a different A^. Note that the operator A^TA comes up naturally when one considers what the action of A does to the length of vectors: |u|^2 = <u,u> |Au|^2 = <Au,Au> = <u,A^Au> Geometrically, the SVD tells us how a linear transformation dilates or contracts space in different directions, e.g., think about what a linear transformation does to a sphere. But all these concepts -- length of vectors, spheres, ellipsoids (images of spheres under linear transformations) -- depend on a choice of inner product.
jacobolus 6y ago
The SVD breaks a linear transformation into 3 parts: (1) A rotation in the source space, (2) an axis-aligned non-uniform scaling (if necessary adding/removing dimensions to match the destination space), and (3) another rotation in the destination space. Rotation is only a meaningful concept when you have a metrical structure (think of the full geometry of Euclid’s Elements including circles, perpendicularity, lengths, angles), so if you don’t have a metrical structure the SVD is likewise not really meaningful. Often spaces we deal with using linear algebra only have an affine structure (parallelism is well defined, and lengths can be compared when they are along parallel lines), but do not have a metrical structure (so there is no meaningful way to compare lengths pointed in different directions or measure angles). For example: there is no meaningful concept of the angle between the directions of 5 miles/gallon and 10 miles/gallon, and if you arbitrarily defined one, it would change when you switched from gallons to milliliters or from miles to meters. In practice people still often arbitrarily impose a metrical structure, sometimes based on some heuristic analysis of the data involved, and then use tools like the SVD based on that.
contravariant 6y ago
The really short answer is that (one off) the definition(s) of SVDs uses the transpose, which requires a inner product.