3 ms·
Looking at the paper here: https://arxiv.org/abs/1610.00883 https://arxiv.org/abs/1610.00883 Can anyone tell me what the P R and F "features" mean in table 2?
by daveguy 10y ago
Looking at the paper here: https://arxiv.org/abs/1610.00883 https://arxiv.org/abs/1610.00883
Can anyone tell me what the P R and F "features" mean in table 2?
They use F-score ( https://en.wikipedia.org/wiki/F1_score https://en.wikipedia.org/wiki/F1_score ) which doesn't take into account true negatives at all. Maybe not an issue since they are evaluating the same set of GoodReads quotes across all four.
The unique features contribute to an F-Score improvement of around 1% with baselines in the 50-80% range. That doesn't seem to be a significant improvement and the F-Score is just a within-test percentage measure of how good the technique is at measuring sarcastic vs not sarcastic. Both of their unique features give improvements across all encodings, but they aren't additive and they differ depending on encoding. That may be statistically significant in some sense (regardless that the improvements are around 1%), but I don't see any statistical evaluation.
Table 2 apparently shows baselines for word2vec, but I expect the baselines are different and the improvements are relative to the appropriate baseline.
The model of how the features determine sarcasm is created using SVM ( https://en.wikipedia.org/wiki/Support_vector_machine https://en.wikipedia.org/wiki/Support_vector_machine ).
Can someone with experience in NLP comment on this paper?
- cerrelio 10y agoP = precision R = recall F = F-score F-score is just the harmonic mean of precision and recall. https://en.wikipedia.org/wiki/Precision_and_recall https://en.wikipedia.org/wiki/Precision_and_recall The paper isn't that extraordinary. Sarcasm detection is considered a hard problem. However, if you get a result that's "better" than some other published paper, you usually publish your work. You get (at most) a 5% boost in F-score using embedding features. Word embeddings are easy to work with, so it's not usually difficult to add them in with commonly used NLP features. You can give your model metrics an effortless bump. Also, it looks like there could be a deep learning model that performs better already (https://techxplore.com/news/2016-08-deep-neural-network-approach-sarcasm.html https://techxplore.com/news/2016-08-deep-neural-network-appr...). I've done two deep learning projects at work. I started learning about the techniques this past spring. They're ridiculously good, especially if you have a ton of data. Feature engineering (like in the paper) is usually a laborious process, and it often requires you to be knowledgeable in the problem domain. With neural nets if you have a reasonable architecture and choose sensible data representations (so your backpropagation converges), it can often "just work" without much tweaking.