6 ms·

A Beginner’s Guide to Eigenvectors, Covariance, PCA and Entropy

199 points by vonnik 11y ago

vonnik 11y ago
DL4J co-creator here: we did a tutorial on restricted Boltzmann machines last week that people might find helpful as well: http://deeplearning4j.org/restrictedboltzmannmachine.html http://deeplearning4j.org/restrictedboltzmannmachine.html
tsumnia 11y ago
You linked it on the bottom of the page, but how do you think the article compares with http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf http://www.cs.otago.ac.nz/cosc453/student_tutorials/principa... ? I'm planning to read through it a little more this weekend, but I remember using the PDF and a whiteboard back in the day to understand PCA.
- vonnik 11y ago
  The PDF through the link is excellent. Two thumbs up.
  - deleted 11y ago
    [deleted]
bcheung 11y ago
Very simple to understand and straight forward. Thanks for writing this up.
tptacek 11y ago
This is extraordinarily well written.
jcadam 11y ago
Nice. I was futzing with covariance matrices in Scala today at work and realized I really need a refresher on some of this stuff (college was over a decade ago). Fortunately, we have a couple of scientists on staff I can usually bug for assistance when I get confused.
abhisheks91 11y ago
Very well written !! Thank you !
alephone 11y ago
"square matrices have as many eigenvectors as they have linearly independent dimensions; i.e. a 2 x 2 matrix would have two eigenvectors, a 3 x 3 matrix three, and an n x n matrix would have n eigenvectors, each one representing its line of action in one dimension." This is not quite right. First of all, if v is an eigenvector of A, then a*v for any non-zero a is also an eigenvector of A, so it makes no sense to say "there are n eigenvectors". Perhaps you meant to say n linearly independent eigenvectors? But this is also not correct. An n x n matrix does not necessarily have to have n linearly independent eigenvectors; consider for instance A = [1, 1; 0, 1], which has dim N(A - I) = 1 (hence 1 linearly independent eigenvector). Basically, not every matrix is diagonalizable. This example also shows that rank(A) is also not necessarily equal to the number of linearly independent eigenvectors (is that what you meant by linearly independent dimensions?). A more correct statement would be to say every symmetric matrix has n linearly independent eigenvectors (and its eigenvectors can be chosen to form an orthonormal basis for R^n).
- vonnik 11y ago
  Thanks for making that point. The statement "square matrices have as many eigenvectors as they have linearly independent dimensions" is meant to inform the statements that follow.
  - jordigh 11y ago
    This statement makes no sense. Independence is not a property of dimensions. A set of vectors is said to be independent if the only linear combination of them that is zero is the trivial linear combination. Otherwise, they are dependent. Or do you have a definition of independence for dimensions? That would first require a different definition of dimension. Ordinarily, dimension is just the cardinality of an independent, spanning set (we call this a basis). It is a single number. A linear space does not have a plurality of dimensions, nor do matrices have an associated plurality of dimensions. Please don't mix technical words without defining them. In other areas of life, humans get along fine by using very imprecise, poetic, ambiguous, and metaphoric language. This is not how mathematics works. Everything you say must have a precise definition. :-(
bigger_cheese 11y ago
Thank you for posting this. It was fortuitously timed I am reading through a thesis at the moment where the author has used PCA along with something called a PLS (Projection to Latent Structure) to perform a multivariate analysis. I was hoping I could use this technique on my own dataset I was in the process of googling the acronyms when I stumbled on this link in HN.
madvoid 11y ago
I understand the concept of eigenvectors/eigenvalues, and I understand the concept of the covariance matrix, but I'm having a hard time understanding why the eigenvectors of the covariance matrix represent the principal components. Does anyone have a intuitive explanation they would like to share? Maybe I don't understand eigenvectors as well as I thought I did...
- tanderson92 11y ago
  The eigenvectors of any diagonalizable matrix (which any covariance matrix is, see elsewhere on this page for a discussion of why this is the case) form a basis for the space in which the matrix operates. The eigenvectors corresponding to the "largest" eigenvalues are called principal components because they are the directions in which the matrix principally act. Consider a matrix A which has two eigenvectors u, v with eigenvalues a, b, and a >> b. Then A(u + v) = au + bv. But since a >> b the operator is stretching u moreso than v, and as a result we think of an operator acting 'principally' in direction u rather than v, and hence the eigenvectors are considered the 'principal components'. I exaggerated the case by letting a >> b, but the same holds true for a similar in magnitude to b, but by considering multiple applications of the matrix you see the same effect: A^n (u + v). covariance matrices are nothing special in this regard.
- thearn4 11y ago
  Personally, I think an explanation using the SVD would probably be more intuitive: http://math.stackexchange.com/questions/3869/what-is-the-intuitive-relationship-between-svd-and-pca http://math.stackexchange.com/questions/3869/what-is-the-int...
- zmmmmm 11y ago
  Here is a totally intuitive and probably wrong way to think about it, but its what I have in my head as my own mental model. Imagine that a matrix transformation is like a wind blowing in a particular direction. If you throw a ball, it gets blown by the wind, and its direction gets changed. If you throw it obliquely it ends up a certain distance from you. Which direction would you throw it in to make it go as far as possible? You'd line up the throw with the direction of the wind. Effectively that's finding your largest eigenvalue / eigenvector. Now, for finding the first principle component we do something special. We create a wind that blows in the direction of variation of the data. We do it by getting the covariance between all our variables and substituting them into the each position of a matrix so that they make up vectors where each dimension is weighted according to the size of the variation in that direction. This makes a "wind" in the direction of the variation of the data. The first principle component finds the vector that most aligns with this covariance "wind". I am sure this is technically wrong in essential ways and I'd love to hear it be corrected!
stared 11y ago
This "entropy" part is missing - no pictures, no formulae... Delete it or elaborate carefully. (Other parts are great!)

199 points by vonnik 11y ago

DL4J co-creator here: we did a tutorial on restricted Boltzmann machines last week that people might find helpful as well: http://deeplearning4j.org/restrictedboltzmannmachine.html http://deeplearning4j.org/restrictedboltzmannmachine.html

tsumnia 11y ago

You linked it on the bottom of the page, but how do you think the article compares with http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf http://www.cs.otago.ac.nz/cosc453/student_tutorials/principa... ? I'm planning to read through it a little more this weekend, but I remember using the PDF and a whiteboard back in the day to understand PCA.

vonnik 11y ago
The PDF through the link is excellent. Two thumbs up.
- deleted 11y ago
  [deleted]

bcheung 11y ago

Very simple to understand and straight forward. Thanks for writing this up.

tptacek 11y ago

This is extraordinarily well written.

jcadam 11y ago

Nice. I was futzing with covariance matrices in Scala today at work and realized I really need a refresher on some of this stuff (college was over a decade ago). Fortunately, we have a couple of scientists on staff I can usually bug for assistance when I get confused.

abhisheks91 11y ago

Very well written !! Thank you !

alephone 11y ago

"square matrices have as many eigenvectors as they have linearly independent dimensions; i.e. a 2 x 2 matrix would have two eigenvectors, a 3 x 3 matrix three, and an n x n matrix would have n eigenvectors, each one representing its line of action in one dimension." This is not quite right. First of all, if v is an eigenvector of A, then a*v for any non-zero a is also an eigenvector of A, so it makes no sense to say "there are n eigenvectors". Perhaps you meant to say n linearly independent eigenvectors? But this is also not correct. An n x n matrix does not necessarily have to have n linearly independent eigenvectors; consider for instance A = [1, 1; 0, 1], which has dim N(A - I) = 1 (hence 1 linearly independent eigenvector). Basically, not every matrix is diagonalizable. This example also shows that rank(A) is also not necessarily equal to the number of linearly independent eigenvectors (is that what you meant by linearly independent dimensions?). A more correct statement would be to say every symmetric matrix has n linearly independent eigenvectors (and its eigenvectors can be chosen to form an orthonormal basis for R^n).

vonnik 11y ago
Thanks for making that point. The statement "square matrices have as many eigenvectors as they have linearly independent dimensions" is meant to inform the statements that follow.
- jordigh 11y ago
  This statement makes no sense. Independence is not a property of dimensions. A set of vectors is said to be independent if the only linear combination of them that is zero is the trivial linear combination. Otherwise, they are dependent. Or do you have a definition of independence for dimensions? That would first require a different definition of dimension. Ordinarily, dimension is just the cardinality of an independent, spanning set (we call this a basis). It is a single number. A linear space does not have a plurality of dimensions, nor do matrices have an associated plurality of dimensions. Please don't mix technical words without defining them. In other areas of life, humans get along fine by using very imprecise, poetic, ambiguous, and metaphoric language. This is not how mathematics works. Everything you say must have a precise definition. :-(

bigger_cheese 11y ago

Thank you for posting this. It was fortuitously timed I am reading through a thesis at the moment where the author has used PCA along with something called a PLS (Projection to Latent Structure) to perform a multivariate analysis. I was hoping I could use this technique on my own dataset I was in the process of googling the acronyms when I stumbled on this link in HN.

madvoid 11y ago

I understand the concept of eigenvectors/eigenvalues, and I understand the concept of the covariance matrix, but I'm having a hard time understanding why the eigenvectors of the covariance matrix represent the principal components. Does anyone have a intuitive explanation they would like to share? Maybe I don't understand eigenvectors as well as I thought I did...

tanderson92 11y ago
The eigenvectors of any diagonalizable matrix (which any covariance matrix is, see elsewhere on this page for a discussion of why this is the case) form a basis for the space in which the matrix operates. The eigenvectors corresponding to the "largest" eigenvalues are called principal components because they are the directions in which the matrix principally act. Consider a matrix A which has two eigenvectors u, v with eigenvalues a, b, and a >> b. Then A(u + v) = au + bv. But since a >> b the operator is stretching u moreso than v, and as a result we think of an operator acting 'principally' in direction u rather than v, and hence the eigenvectors are considered the 'principal components'. I exaggerated the case by letting a >> b, but the same holds true for a similar in magnitude to b, but by considering multiple applications of the matrix you see the same effect: A^n (u + v). covariance matrices are nothing special in this regard.
thearn4 11y ago
Personally, I think an explanation using the SVD would probably be more intuitive: http://math.stackexchange.com/questions/3869/what-is-the-intuitive-relationship-between-svd-and-pca http://math.stackexchange.com/questions/3869/what-is-the-int...
zmmmmm 11y ago
Here is a totally intuitive and probably wrong way to think about it, but its what I have in my head as my own mental model. Imagine that a matrix transformation is like a wind blowing in a particular direction. If you throw a ball, it gets blown by the wind, and its direction gets changed. If you throw it obliquely it ends up a certain distance from you. Which direction would you throw it in to make it go as far as possible? You'd line up the throw with the direction of the wind. Effectively that's finding your largest eigenvalue / eigenvector. Now, for finding the first principle component we do something special. We create a wind that blows in the direction of variation of the data. We do it by getting the covariance between all our variables and substituting them into the each position of a matrix so that they make up vectors where each dimension is weighted according to the size of the variation in that direction. This makes a "wind" in the direction of the variation of the data. The first principle component finds the vector that most aligns with this covariance "wind". I am sure this is technically wrong in essential ways and I'd love to hear it be corrected!

stared 11y ago

This "entropy" part is missing - no pictures, no formulae... Delete it or elaborate carefully. (Other parts are great!)