3 ms·
If you weight these counts as you describe, your sentiment for this sentence will be -0.44 + 1 = 0.66 even though the sentence is clearly 'neutral' and should h
by danieldk 12y ago
If you weight these counts as you describe, your sentiment for this sentence will be -0.44 + 1 = 0.66 even though the sentence is clearly 'neutral' and should have a score of 0.
If you want to stick to simple counting (it's a fun exercise at least ;)) and L is the large lexicon and S the small, why don't you:
- Generate L' by randomly picking |S| words from L.
- Compute the score using L' and S.
- Rinse and repeat for the same text N times.
- Compute an aggregate over the N scores, e.g. average, score with the largest number of hits from L, score with the largest number of hits from both, ...
This way, the lexicons have the same size during each scoring attempt, but you do use the extra vocabulary of the larger lexicon.
Ps.: don't stick to simple counting. It doesn't work ;).
- markovbling 12y agoAhh resampling |S| words from L is a great idea! :) I know simple counting not the greatest approach but I started out by trying to replicate a research paper put out by a stock broker (not the most advanced research haha!) I would love some suggestions for a method that does work :)
- markovbling 12y agoOn second thought, not sure resampling |S| words from L is a good idea because you want 100% coverage of your sentiment universe and an asymmetrical corpus is not a priori incorrect so resampling does not solve anything except reweighting match lists which may not even lead to symmetrical sentiment since the distribution of words amongst the language (TF-IDF helps here) is not necessarily the same in your negative and positive lists