3 ms·
Yeah, I realise what I suggested was a hack. But simple word counting is so error prone that the real answer is not to use it. I am curious if the word 'not' w
by peterhi 12y ago
Yeah, I realise what I suggested was a hack. But simple word counting is so error prone that the real answer is not to use it.
I am curious if the word 'not' was in your negative list, it is quite a difficult word to handle. "I was not disappointed" against "I will not do that again".
Have you looked to see if you are having runs of negative terms? "bad" counts as 1, "fucking awful" counts as 2. If you are getting runs of negative terms perhaps you might score them as 1/1 for the first term in a run, 1/2 for the second term, 1/3 down to 1/n. Of course given these are journals then perhaps such phrases are unlikely to occur :) Unlike on twitter and the comment section of blogs.
Could it be that the style of writing is producing this bias? I am reading a book on statistical inference and it is continually pointing out how not to use the techniques because they could lead to erroneous conclusions. I suspect that it would score badly with simple word counting.
- markovbling 12y agoHaha! :) Totally agree - definitely need to do something about negation e.g. "Not bad" != "bad" My understanding is that this is usually handled using a list of adverbs e.g. 'not' / 'very' ('very bad' > 'bad') etc. Not sure of a better approach than word counting though?
- danieldk 12y agoNot sure of a better approach than word counting though? There are many better approaches, assuming that you have annotations for supervised learning. E.g.: http://www.socher.org/uploads/Main/SocherPenningtonHuangNgManning_EMNLP2011.pdf http://www.socher.org/uploads/Main/SocherPenningtonHuangNgMa... http://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf http://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf