The eigenvectors of any diagonalizable matrix (which any covariance matrix is, see elsewhere on this page for a discussion of why this is the case) form a basis

The eigenvectors of any diagonalizable matrix (which any covariance matrix is, see elsewhere on this page for a discussion of why this is the case) form a basis for the space in which the matrix operates. The eigenvectors corresponding to the "largest" eigenvalues are called principal components because they are the directions in which the matrix principally act. Consider a matrix A which has two eigenvectors u, v with eigenvalues a, b, and a >> b. Then A(u + v) = au + bv. But since a >> b the operator is stretching u moreso than v, and as a result we think of an operator acting 'principally' in direction u rather than v, and hence the eigenvectors are considered the 'principal components'. I exaggerated the case by letting a >> b, but the same holds true for a similar in magnitude to b, but by considering multiple applications of the matrix you see the same effect: A^n (u + v). covariance matrices are nothing special in this regard.