3 ms·

Just piping in to say that I'm very curious as well! How do you go from word vectors to a description of the document?

useracd 9y ago
Often the w2v embedding layer will be the first layer of a network, through which a document of word representations will be passed. The output of the embedding layer will be a 2 dimensional tensor with the embedding dimension in one direction and the number of input words in the other. The next step is often to apply zero or more convolutional units over the word direction before some kind of pooling to get a 1 dimensional output. The output of the pooling layer will be a document representation based upon the word vectors. Other approaches like doc2vec are also used.
- ppod 9y ago
  Thank you for this very helpful information, and I have just one more question: "Often the w2v embedding layer will be the first layer of a network, through which a document of word representations will be passed. The output of the embedding layer will be a 2 dimensional tensor with the embedding dimension in one direction and the number of input words in the other. The next step is often to apply zero or more convolutional units over the word direction before some kind of pooling to get a 1 dimensional output. The output of the pooling layer will be a document representation based upon the word vectors. Other approaches like doc2vec are also used." I thought the approach you described was doc2vec. If not, then does it have a name/citation?
  - aglionby 9y ago
    Doc2vec is the name of the gensim implementation of this paper [0]. Briefly, it creates document embeddings (document being anything on the level of phrases, to sentences, to paragraphs and beyond) on the basis of either predicting a word given the context words (distributed memory -- DM), or predicting the context words given a word (distributed bag of words -- DBOW). [0] https://arxiv.org/abs/1405.4053 https://arxiv.org/abs/1405.4053
- isoprophlex 9y ago
  So in simple terms, you concatenate the word vectors and feed them to further network layers to arrive at a reduced-D representation? Either way... Very nice, thanks!
IanCal 9y ago
The simplest thing, and one that apparently works fairly well with fasttext vectors, is just to average them.
- isoprophlex 9y ago
  I'm surprised (and delighted) you say that such a straightforward approach works well, thanks.
aglionby 9y ago
Combining word embeddings to get a single representation of a sentence in general is a pretty hard problem. Point-wise averaging, addition or summation [0] works alright, but it's not great. Some more recent work looks a using RNNs to combine them; most of the literature I've seen is pretty application-specific, but try [1][2] for flavour. [0] http://www.aclweb.org/anthology/P/P08/P08-1028 http://www.aclweb.org/anthology/P/P08/P08-1028 [1] http://arxiv.org/abs/1409.3215 http://arxiv.org/abs/1409.3215 [2] http://arxiv.org/abs/1409.0473 http://arxiv.org/abs/1409.0473
- homarp 9y ago
  [0] link should be http://www.aclweb.org/anthology/P/P08/P08-1028.pdf http://www.aclweb.org/anthology/P/P08/P08-1028.pdf Jeff Mitchell; Mirella Lapata - Vector-based Models of Semantic Composition

Often the w2v embedding layer will be the first layer of a network, through which a document of word representations will be passed. The output of the embedding layer will be a 2 dimensional tensor with the embedding dimension in one direction and the number of input words in the other. The next step is often to apply zero or more convolutional units over the word direction before some kind of pooling to get a 1 dimensional output. The output of the pooling layer will be a document representation based upon the word vectors. Other approaches like doc2vec are also used.

ppod 9y ago
Thank you for this very helpful information, and I have just one more question: "Often the w2v embedding layer will be the first layer of a network, through which a document of word representations will be passed. The output of the embedding layer will be a 2 dimensional tensor with the embedding dimension in one direction and the number of input words in the other. The next step is often to apply zero or more convolutional units over the word direction before some kind of pooling to get a 1 dimensional output. The output of the pooling layer will be a document representation based upon the word vectors. Other approaches like doc2vec are also used." I thought the approach you described was doc2vec. If not, then does it have a name/citation?
- aglionby 9y ago
  Doc2vec is the name of the gensim implementation of this paper [0]. Briefly, it creates document embeddings (document being anything on the level of phrases, to sentences, to paragraphs and beyond) on the basis of either predicting a word given the context words (distributed memory -- DM), or predicting the context words given a word (distributed bag of words -- DBOW). [0] https://arxiv.org/abs/1405.4053 https://arxiv.org/abs/1405.4053
isoprophlex 9y ago
So in simple terms, you concatenate the word vectors and feed them to further network layers to arrive at a reduced-D representation? Either way... Very nice, thanks!

IanCal 9y ago

The simplest thing, and one that apparently works fairly well with fasttext vectors, is just to average them.

isoprophlex 9y ago
I'm surprised (and delighted) you say that such a straightforward approach works well, thanks.

aglionby 9y ago

Combining word embeddings to get a single representation of a sentence in general is a pretty hard problem. Point-wise averaging, addition or summation [0] works alright, but it's not great. Some more recent work looks a using RNNs to combine them; most of the literature I've seen is pretty application-specific, but try [1][2] for flavour. [0] http://www.aclweb.org/anthology/P/P08/P08-1028 http://www.aclweb.org/anthology/P/P08/P08-1028 [1] http://arxiv.org/abs/1409.3215 http://arxiv.org/abs/1409.3215 [2] http://arxiv.org/abs/1409.0473 http://arxiv.org/abs/1409.0473

homarp 9y ago
[0] link should be http://www.aclweb.org/anthology/P/P08/P08-1028.pdf http://www.aclweb.org/anthology/P/P08/P08-1028.pdf Jeff Mitchell; Mirella Lapata - Vector-based Models of Semantic Composition