4 ms·
You may be painting with too broad a brush. I've worked with a number of NLP people to try to beat our tfidf model using deep methods and various word vectors.
by mooneater 9y ago
You may be painting with too broad a brush.
I've worked with a number of NLP people to try to beat our tfidf model using deep methods and various word vectors. The tfidf still performs better.
I think the reasons are: many out-of-vocab words, and small amounts of labelled data.
- PeterisP 9y agoComing a bit late to the discussion, but out-of-vocab words and small amounts of labelled data is the exact kind of problem that word embedding vectors are meant to solve, and they do so exceedingly well. Of course, you need a good, large source of (unlabelled) data, preferably approaching the gold standard of "everything that has ever been written about your problem domain" to train the embeddings, and that takes some effort, but then the out-of-vocab words must be exceedingly rare (i.e. only if it's a genuinely new word that has never ever been mentioned before, or a very very rare term that you've discarded), and even that can be solved well with character-based embeddings.