4 ms·

Practical Text Classification with Python and Keras

60 points by endlesstrax 8y ago

ma2rten 8y ago
Neural networks often perform worse than bag of words for text classification. They compared to a bag of words baseline, but they didn't tune the regularization parameter, which is important. They should probably have also tried tf-idf and n-grams.
- ScoutOrgo 8y ago
  Definitely not true. Transfer learning a model trained on a corpus then fine tuned on labeled classes is current SotA approach [1]. [1] http://nlp.fast.ai/category/classification.html http://nlp.fast.ai/category/classification.html
  - andreasvc 8y ago
    You mention a single counterexample, but is it "definitely not true"? I think there is a strong publication bias: papers will report when they improve on the baseline, but that's likely not representative for the common case (e.g., limited data, no pretrained model available, no time for extensive parameter tuning).
- thanatropism 8y ago
  I'm ignorant. I thought "bag of words" was a representation (the idea that a sentence has no grammar, it's just represented by its words as columns, maybe weighted by tf-idf who knows), not a specific model. Where can I read about this bag of words?
  - andreasvc 8y ago
    You're right that it is a representation, but also an instance of the vector space model of language. Coupled with a linear model for prediction it is a strong baseline for text classification problems. See e.g. http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html http://scikit-learn.org/stable/tutorial/text_analytics/worki...
    - thanatropism 8y ago
      So "bag of words" = "count/tf-idf vectorizer + logistic/ridge/lasso regression"? Also: a vector space is a set of things that can be added and multiplied by a scalar. So a vector space model should be the proverbial representation where "queen - woman + man = king". Am I being an insufferable pedant? I follow text analysis only very lightly and keep losing the thread.
      - andreasvc 8y ago
        Yes, in the context of text classification a bag of words model will refer to that, or combined with some other linear model like linear SVM or naive bayes. The queen - woman example is when you try to make a model of word semantics, such as with word2vec. In a document classification task the vectors represent documents.

60 points by endlesstrax 8y ago

Neural networks often perform worse than bag of words for text classification. They compared to a bag of words baseline, but they didn't tune the regularization parameter, which is important. They should probably have also tried tf-idf and n-grams.

ScoutOrgo 8y ago
Definitely not true. Transfer learning a model trained on a corpus then fine tuned on labeled classes is current SotA approach [1]. [1] http://nlp.fast.ai/category/classification.html http://nlp.fast.ai/category/classification.html
- andreasvc 8y ago
  You mention a single counterexample, but is it "definitely not true"? I think there is a strong publication bias: papers will report when they improve on the baseline, but that's likely not representative for the common case (e.g., limited data, no pretrained model available, no time for extensive parameter tuning).
thanatropism 8y ago
I'm ignorant. I thought "bag of words" was a representation (the idea that a sentence has no grammar, it's just represented by its words as columns, maybe weighted by tf-idf who knows), not a specific model. Where can I read about this bag of words?
- andreasvc 8y ago
  You're right that it is a representation, but also an instance of the vector space model of language. Coupled with a linear model for prediction it is a strong baseline for text classification problems. See e.g. http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html http://scikit-learn.org/stable/tutorial/text_analytics/worki...
  - thanatropism 8y ago
    So "bag of words" = "count/tf-idf vectorizer + logistic/ridge/lasso regression"? Also: a vector space is a set of things that can be added and multiplied by a scalar. So a vector space model should be the proverbial representation where "queen - woman + man = king". Am I being an insufferable pedant? I follow text analysis only very lightly and keep losing the thread.
    - andreasvc 8y ago
      Yes, in the context of text classification a bag of words model will refer to that, or combined with some other linear model like linear SVM or naive bayes. The queen - woman example is when you try to make a model of word semantics, such as with word2vec. In a document classification task the vectors represent documents.