4 ms·
This is great for quick jobs but if you have a good dataset that is in your domain you should retrain using fasttext, doc2vecC, word2vec or starspace to name bu
by visarga 9y ago
This is great for quick jobs but if you have a good dataset that is in your domain you should retrain using fasttext, doc2vecC, word2vec or starspace to name but a few tools that are excellent. The window size is important as well, depending on your downstream task - one size doesn't fit all. It only takes a few hours on a beefy CPU with enough RAM for a corpus with a size of billions of words.
Word vectors are fascinating representations. There is a huge amount of information and nuance captured in them. You can use them directly for topic retrieval (using annoy or another optimised vector index), or feed them into a classifier such as those in the Sklearn library. All types of neural nets: fully connected, recurrent and convolutional can be applied on word vectors.
- rspeer 9y agoI argue that you need both domain-general and domain-specific data. Most domains you could apply NLP to don't have billions of words to train on, especially if the language of the data isn't English. Pre-trained data is valuable, and you really aren't going to re-learn it all from your data, so why throw it out?
- laughingman2 9y agoYes, recent work on transfer learning https://arxiv.org/abs/1801.06146 https://arxiv.org/abs/1801.06146 " Fine-tuned Language Models for Text Classification" suggests that training a lstm on a large corpora for language modeling, then using it as sentence encoder after tuning it in specific dataset gives a considerable perfomance. This is covered in fast.ai 's new course 1.