3 ms·
Great idea! I've looked at term-weighting approaches such as TF-IDF but I don't have a training set of positive / negative sentences so would have to term-weigh
by markovbling 12y ago
Great idea! I've looked at term-weighting approaches such as TF-IDF but I don't have a training set of positive / negative sentences so would have to term-weight just the occurances of each of positive/sentiment list and compute a net sentiment on this basis.
Will implement and see if this fixes the bias introduced by asymmetrical corpus sizes.
Fundamentally, I'm not sure there is a 'solution' available or even necessary to the issue of a bigger negative word list than positive word list. So what if there are more ways of saying negative things, all that matters is how many times positive or negative things are said. My problem then, however, is even if your range of terms that can create a match is greater for negative words, you should still have representative positive/negative counts and that doesn't explain why I'm getting much more negative sentiment scores than would be expected given my test documents.
- tfgg 12y agoI think the issue is whether P(detected|positive) and P(detected|negative) are the same, right? It's whether they have equal coverage of all possible phrases. Having more positive or negative words, as you say, doesn't inherently bias things. Is your hypothesis that your corpus doesn't skew negative, which seems to be the basis of this question, correct? Can you do some manual sampling to get a good bound on it?
- markovbling 12y agoTotally agree I suspect the bias currently giving me P(detected|negative)>P(detected|positive) is resulting from my simplification to looking only at 1-grams