4 ms·
The baseline I'd like to see this compared to is the not-very-deep-learning "bag of tricks" that's conveniently implemented in fastText [1]. [1] https://github
by rspeer 10y ago
The baseline I'd like to see this compared to is the not-very-deep-learning "bag of tricks" that's conveniently implemented in fastText [1].
[1] https://github.com/facebookresearch/fastText https://github.com/facebookresearch/fastText
- bowlesbe 10y agoGreat point! I considering using fasttext as a baseline, however in practice fasttext really didn't work well at all with the small data set, much worse than the tfidf baseline. I think Fasttext's classification approach might not work well with such a small dataset. I'm not sure but I suspect its because it tries to learn embeddings - but there just isn't anywhere near enough data for that. I'd love an outside perspective on this.
- rspeer 10y agoFair enough. That's a useful comparison to know about. But I'm wondering how you get around that with the neural net. In the post, you said there are only a few hundred labeled examples, right? How can a neural net with hundreds of parameters set those parameters to anything reasonable, and not overfit, when there are about as many parameters as examples?
- bowlesbe 10y agoGreat question and I share your intuition but I think its all properly regularizing your model. I guess for neural networks, Dropout works really darn well as a regularization strategy. I could have tried to see whether performance dropped significantly without dropout.
- botexpert 10y agowhy not vowpal wabbit? fast text is practically a ad-hoc version of mathematically proven vowpal wabbit?
- rspeer 10y agoI've used both. fastText will openly tell you that it's basically Vowpal Wabbit but faster (hence the name), with a way to stick word embeddings in without pre-processing, and with fewer ways to shoot yourself in the foot.