7 ms·

For the CNN model I was using Magpie [0] which takes a sum of the vectors (so yes, equivalent to unweighted average) to represent the document. For the FastTex

by wenc 9y ago

For the CNN model I was using Magpie [0] which takes a sum of the vectors (so yes, equivalent to unweighted average) to represent the document. For the FastText model, it would be whatever FastText is doing under the hood. I haven't peeked at the code to see what it is doing, but Section 2 of the paper I cited above seems to imply that an average is taken. [0] https://github.com/inspirehep/magpie https://github.com/inspirehep/magpie

useracd 9y ago
You're correct! The keras example implementation of FastText uses a global average pooling is used to reduce the dimensionality [0]. [0] https://github.com/keras-team/keras/blob/master/examples/imdb_fasttext.py#L123 https://github.com/keras-team/keras/blob/master/examples/imd...
ppod 9y ago
Interesting. I had heard specifically that word vectors weren't a game-changer for document classification, because the averaging method didn't work well. But the difference between word2vec and FastText must be important. I think your baseline of Naive Bayes + TF-IDF would do a lot better if it was SVM and TF-IDF, but I didn't expect the big jump between word2vec and FastText.
- wenc 9y ago
  > I had heard specifically that word vectors weren't a game-changer for document classification, because the averaging method didn't work well. As with anything, your mileage may vary. One aspect of FastText that definitely helped in my case was n-gram support (both word and character, tunable via command-line arguments). In my corpus, I have short fragments of sentences containing misspelled words, incorrect grammar etc. plus my test set has out-of-vocabulary words. n-grams are more robust to these than Word2Vec which uses a static vocabulary.
  - swsieber 9y ago
    I thanked your top level comment, but this whole thread is great. I fond my self worth data that sounds like yours. I'm excited to try fast text.
- sp332 9y ago
  I think it's important to use the right "average" operation. word2vec puts words on the surface of a hypersphere, so "adding" vectors means turning around the sphere.

by wenc 9y ago

useracd 9y ago

You're correct! The keras example implementation of FastText uses a global average pooling is used to reduce the dimensionality [0]. [0] https://github.com/keras-team/keras/blob/master/examples/imdb_fasttext.py#L123 https://github.com/keras-team/keras/blob/master/examples/imd...

ppod 9y ago

Interesting. I had heard specifically that word vectors weren't a game-changer for document classification, because the averaging method didn't work well. But the difference between word2vec and FastText must be important. I think your baseline of Naive Bayes + TF-IDF would do a lot better if it was SVM and TF-IDF, but I didn't expect the big jump between word2vec and FastText.

wenc 9y ago
> I had heard specifically that word vectors weren't a game-changer for document classification, because the averaging method didn't work well. As with anything, your mileage may vary. One aspect of FastText that definitely helped in my case was n-gram support (both word and character, tunable via command-line arguments). In my corpus, I have short fragments of sentences containing misspelled words, incorrect grammar etc. plus my test set has out-of-vocabulary words. n-grams are more robust to these than Word2Vec which uses a static vocabulary.
- swsieber 9y ago
  I thanked your top level comment, but this whole thread is great. I fond my self worth data that sounds like yours. I'm excited to try fast text.
sp332 9y ago
I think it's important to use the right "average" operation. word2vec puts words on the surface of a hypersphere, so "adding" vectors means turning around the sphere.