> I am also very interested in knowing what to do when the dimension of the input space can vary. It is frequently the case that one's data lies on a (curvy) l

> I am also very interested in knowing what to do when the dimension of the input space can vary. It is frequently the case that one's data lies on a (curvy) low dimensional manifold embedded within a higher dimensional space. Because of this, performing an orthogonal projection to a lower dimension is often a bad idea. Consider the famous "Swiss roll" dataset: http://www.convexoptimization.com/dattorro/manifold_learning.html http://www.convexoptimization.com/dattorro/manifold_learning... Although the data lies on a 2D manifold, there exists no affine subspace to project the data onto; one must instead perform a non-linear mapping. Problems like this are tackled using a variety of methods that attempt to "unroll" the manifold into its own vector space. The rest of the problem can then be solved using the tools of linear algebra. See http://repository.upenn.edu/cgi/viewcontent.cgi?article=1001&context=cis_papers http://repository.upenn.edu/cgi/viewcontent.cgi?article=1001... for example.