The best part about this is how they calculated happiness from each tweet: http://arxiv.org/pdf/1101.5120.pdf http://arxiv.org/pdf/1101.5120.pdf > we avoid ste

The best part about this is how they calculated happiness from each tweet: http://arxiv.org/pdf/1101.5120.pdf http://arxiv.org/pdf/1101.5120.pdf > we avoid stemming words, i.e., conflating inflected words with their root form, such as all conjugations of a specic verb. For verbs in particular, by focusing on the most fre- quent words, we obtained scores for those conjuga- tions likely to appear in texts, obviating any need for stemming. Moreover, while we observe stemming works well in some cases for happiness measures, e.g., havg(advance)=6.58, havg(advanced)=6.58, and havg(advances)=6.24, it fails badly in others, e.g., havg(have)=5.82 and havg(had)=4.74; havg(arm)=5.50 and havg(armed)=3.84; and havg(capture)=4.18 and havg(captured)=3.22. That is something I came across myself when I was working on a similar data mining project. But this was a surprise: > In terms of methodology, our hedonometer could be improved by incorporating happiness estimates for common n-grams, e.g., 2-grams such as `child abuse' and `sex scandal' as well as negated sentiments such as `not happy' I can understand ignoring n-grams but a simple no-X or not-X is a necessity. In common parlance, 'Peace' > 'No war' > 'No peace' > 'War'. If you assume the simple happiness values of 'Peace' = 1, 'War' = -1, 'No' = -0.5, then the scores end up: Peace=1, No War=-1.5, No peace=0.5, War=-1. Clearly 'War' is worse than 'No War' yet the scores don't correspond. The correct order should be something like: Peace=1, No War=0.5, No peace=-0.5, War=-1. To make this work, 'No' shouldn't be a fixed negative value like -0.5. It should be -(next-word/2). Then 'No Peace' = -0.5 + 1 = 0.5 and 'No War' = 0.5 + -1 = -0.5. PS: I typed 'War' so many times in the above sentence that I had to check its spelling to confirm it was a real word. http://en.wikipedia.org/wiki/Semantic_satiation http://en.wikipedia.org/wiki/Semantic_satiation strikes again.