One of the most famous word embedding methods, Word2Vec, actually approximates word co-occurrence PMI matrix through matrix decomposition. Check out Omer Levy a

One of the most famous word embedding methods, Word2Vec, actually approximates word co-occurrence PMI matrix through matrix decomposition. Check out Omer Levy and Yoav Goldberg's work. In that case given enough dimensions it could approximate any matrix well - i.e. linearity shouldn't be a limiting factor. Correlation (vs causation) and linearity are orthogonal concepts.