4 ms·
If I understood correctly, you are trying to get a sentiment that is always correct for single sentences but that can extrapolate word frequencies if they don't
by QuantumRoar 12y ago
If I understood correctly, you are trying to get a sentiment that is always correct for single sentences but that can extrapolate word frequencies if they don't appear in your list. I.e. for large neutral documents you want it to be neutral, although your negative words match statistically more often.
My intuition tells me that you can't really do both: Either use only those words in your dictionary and get the behaviour right for single sentences, or extrapolate as if both sets where the same size (weighted average, would be the easiest). By extrapolating you may assume, that for each positive match that you get, you'll miss other positive matches. That means you generally underestimate positive matches, compared to negative matches. This only works on large datasets.
But really, how bad is it to get a sentiment of -0.17 for a single sentence? It tells you that it was a negative sentence but that you have a high chance that there was a positive word in there that you missed, which is what you need to implement to get neutral sentiment for large neutral documents.