4 ms·
Good training data's partly hard to come by because there's often reasonably poor inter-annotator agreement on sentiment datasets -- that is to say, humans disa
by jlees 12y ago
Good training data's partly hard to come by because there's often reasonably poor inter-annotator agreement on sentiment datasets -- that is to say, humans disagree a lot in how we interpret a phrase. What reads like sarcasm to you might read like genuine enthusiasm to another.
It's pretty easy to load up a set of data into a crowdsourcing tool and use microtasks to rate it, but my experiences doing so weren't superb (even restricting to native English speakers alone).
A better source of data is starred reviews where you have the star rating and the review itself -- these come free with a sentiment rating, although plenty of caveats around normalization. There are lots of places with review systems like this and some (like Yelp) even make the data available: https://www.yelp.com/academic_dataset https://www.yelp.com/academic_dataset
- Profan 12y agoI wasn't aware that yelp provided a dataset, that's very interesting! Since I had this very problem as I was working on using the output from sentiment analysis to modify sentences so to invert the sentiment polarity (positive to negative, negative to positive), the datasets I found were never general enough (movie reviews, many domain specific terms, hard in the text generation step), or had a lot of noise (twitter dataset). Though evaluating the system was very hard, due to the reasons you stated, inter-annotator agreement was beyond terrible. I'll have to look into if other review services expose their data as well, seems appropriate.