4 ms·
Isn't this all based on LSA (Latent Semantic Analysis; e.g. first major paper was by by Landauer back in 1990, and going back to the 1960's)? I implemented LSA
by NKCSS 10y ago
Isn't this all based on LSA (Latent Semantic Analysis; e.g. first major paper was by by Landauer back in 1990, and going back to the 1960's)?
I implemented LSA a few years ago based on these papers and it just seems like LSA/LSI to me..
[Update]
Just found this:
> In this sense we have come full circle to the methods presented earlier that rely on matrix factorization (such as LSA). Where LSA uses SVD to find the best fitting subspace in terms of the co-variances of words across documents, the Glove model explicitly optimizes wordvectors to reflect the likelihood of their co-occurrence. The point, however, stands that Word2Vec is not an entirely novel approach to arrive at word vectors that capture semantic meaning in their substructure.
- twelfthnight 10y agoJust for some extra clarification: With LSA, each document is transformed into a single vector that has the length of the vocabulary. The length of the vocabulary is the number of unique words across all documents. If a word is present in a document, it is represented as a 1 in the vector and 0 if it is not. So after this transformation, the text is transformed in an D by V matrix where D is the number of documents and V is the vocabulary size. GloVe (and word2vec by equivalence) however, work on a different matrix. In this algorithm, the matrix is V by V, where V is the vocabulary size and each cell counts the number of the times a word appears next to another word in a document. That is, the matrix represents the counts of how often words are neighbors across all documents. (There are some other technicalities, by this the thrust of GloVe). The similarity between LSA and GloVe is that once the respective matrix is built, it is followed by matrix factorization. By matrix factorization, I mean SVD. This is a process which takes an N by M matrix and returns an N by K matrix, essentially squeezing the matrix into fewer columns (K is a free parameter that is smaller than M). So ultimately, what we get with LSA is a D by K matrix (where K is just an arbitrary number, like 100). The interpretation of this matrix is that each row represents a document and each column represents a topic. In GloVe, we get a V by K matrix where each row represents a word and each column represents an embedding.
- jj12345 10y agoAs someone just now getting into the basics of NLP, thank you for this brief summary.
- eginhard 10y agoFrom LSA/SVD you get a V x K matrix as well - that's exactly what the factorisation is doing. The following two papers also go into detail about the mathematical similarities between LSA and neural embeddings and achieving similar performance with both: Levy, O. and Goldberg, Y. (2014). Neural word embedding as implicit matrix factorization. https://www.cs.bgu.ac.il/~yoavg/publications/nips2014pmi.pdf https://www.cs.bgu.ac.il/~yoavg/publications/nips2014pmi.pdf Levy, O., Goldberg, Y., and Dagan, I. (2015). Improving distributional similarity with lessons learned from word embeddings. http://www.anthology.aclweb.org/Q/Q15/Q15-1016.pdf http://www.anthology.aclweb.org/Q/Q15/Q15-1016.pdf
- twelfthnight 10y agoTrue, you could use the right matrix of the SVD on the term document matrix. As far as I know, the embeddings won't have the same interpretation as the left matrix from SVD on the word-context matrix. (By LSA I mean SVD on the term document matrix). Those are excellent papers, by the way.
- pklausler 10y agos/unique/distinct/
- kreetx 10y agoGood summary, thanks!