4 ms·

Socher thinks his model could reach upward of 95 percent accuracy, but it will never be completely perfect. Ridiculous accuracy for something as complex as sen

by hooande 13y ago

Socher thinks his model could reach upward of 95 percent accuracy, but it will never be completely perfect. Ridiculous accuracy for something as complex as sentiment analysis. You don't hear established researchers say something like this often. Moving any number from 85% to 95% is the work of the gods. I wonder if the code they release will include some version of the data from the mechnical turk project. Code for this is great and many people (myself included) will be able to learn a lot from it. But it won't have the same level of reproducibility without the data. If they do release it they will effectively be giving away the money they spent on mechanical turk. 11,000 HITS ain't cheap and they probably had redundant sampling. If they decide to make this data public as well it would be a big win for research because labelled data is so important to machine learning work. Open sourcing the code associated with a research paper is already a huge deal. It's great to see big name researchers like Andrew Ng pushing the trend for publishing code. If nothing else this is a great example for computer science papers going forward.

ninjin 13y ago
> I wonder if the code they release will include some version of the data from the mechnical turk project. I don't know if the "raw" data from MTurk is included in the data set. But at least the finalised data has been released for quite some time now. http://nlp.stanford.edu/~socherr/stanfordSentimentTreebank.zip http://nlp.stanford.edu/~socherr/stanfordSentimentTreebank.z... As for the value of the data, I have heard numbers around 10,000 USD. But I have little more than academic gossip to back these numbers up.
- mrgordon 13y ago
  It is great to see more crowdsourcing data getting open sourced. I haven't read the research but I'm assuming he used a fairly unrestricted crowd. I'd be interested to use the crowd to rate sentiment analysis results from this model vs. CrowdFlower's Senti to see how the model fares against a specialized sentiment analysis crowd. I would be very surprised if the model won. I am willing to run a comparison if there is sufficient interest in the data.
warmfuzzykitten 13y ago
Yes, but a little realism here: They have actually moved it from 80% (using single word analysis) to 85%. That's fantastic, but not the work of the gods.
- nl 13y ago
  I dunno.. Sentiment analysis is often akin to mind reading. I don't know how the average human would do, but given how often people miss jokes & sarcasm on forums & emails it wouldn't surprise me at all to learn that the average human analysis would score below 85% on single sentences without context. (And to be fair to the OP, they said 95% would be the work of gods, and I'd certainly agree that would be better than most humans could do)
- ninjin 13y ago
  I agree that the article is sensational and what really confuses me is that when I read the article and their press release (found below) they are far more humble and down to earth (I really like the writing and the work in general). My personal guess is that this is a classical case of a journalist going overboard (been there...). http://engineering.stanford.edu/news/stanford-algorithm-analyzes-sentence-sentiment-advances-machine-learning http://engineering.stanford.edu/news/stanford-algorithm-anal...
- PeterisP 13y ago
  In general, after the 'easy' gains on a decently researched machine learning problem have been done, a new approach that gets, say, a 2% increase (say, from 91% to 93%) is considered a significant breakthrough and happens rarely - 5-10 years or so. 80%->85% is great. 80%->95% could be called work of the gods, given the 4x reduction in error rate.
adrr 13y ago
Never used Mechanical Turk, 11,000 HITS at 2 to 3 cents(a guess) doesn't seem that pricey. Its the cost of a couple text books. :)
- aantix 13y ago
  Sentiment analysis is something the Turkers can do quickly and efficiently, so those tasks tend to go lower because you can recruit such a larger worker pool for them. Same for image classifications.
- grogenaut 13y ago
  good sentiment or survey results generally run more like $.75-$2
linux_devil 13y ago
Its been there for long time, I don't understand why is he glorifying what Google has achieved in ages. Honestly , I don't find anything new in technique.

by hooande 13y ago

ninjin 13y ago

> I wonder if the code they release will include some version of the data from the mechnical turk project. I don't know if the "raw" data from MTurk is included in the data set. But at least the finalised data has been released for quite some time now. http://nlp.stanford.edu/~socherr/stanfordSentimentTreebank.zip http://nlp.stanford.edu/~socherr/stanfordSentimentTreebank.z... As for the value of the data, I have heard numbers around 10,000 USD. But I have little more than academic gossip to back these numbers up.

mrgordon 13y ago
It is great to see more crowdsourcing data getting open sourced. I haven't read the research but I'm assuming he used a fairly unrestricted crowd. I'd be interested to use the crowd to rate sentiment analysis results from this model vs. CrowdFlower's Senti to see how the model fares against a specialized sentiment analysis crowd. I would be very surprised if the model won. I am willing to run a comparison if there is sufficient interest in the data.

warmfuzzykitten 13y ago

Yes, but a little realism here: They have actually moved it from 80% (using single word analysis) to 85%. That's fantastic, but not the work of the gods.

nl 13y ago
I dunno.. Sentiment analysis is often akin to mind reading. I don't know how the average human would do, but given how often people miss jokes & sarcasm on forums & emails it wouldn't surprise me at all to learn that the average human analysis would score below 85% on single sentences without context. (And to be fair to the OP, they said 95% would be the work of gods, and I'd certainly agree that would be better than most humans could do)
ninjin 13y ago
I agree that the article is sensational and what really confuses me is that when I read the article and their press release (found below) they are far more humble and down to earth (I really like the writing and the work in general). My personal guess is that this is a classical case of a journalist going overboard (been there...). http://engineering.stanford.edu/news/stanford-algorithm-analyzes-sentence-sentiment-advances-machine-learning http://engineering.stanford.edu/news/stanford-algorithm-anal...
PeterisP 13y ago
In general, after the 'easy' gains on a decently researched machine learning problem have been done, a new approach that gets, say, a 2% increase (say, from 91% to 93%) is considered a significant breakthrough and happens rarely - 5-10 years or so. 80%->85% is great. 80%->95% could be called work of the gods, given the 4x reduction in error rate.

adrr 13y ago

Never used Mechanical Turk, 11,000 HITS at 2 to 3 cents(a guess) doesn't seem that pricey. Its the cost of a couple text books. :)

aantix 13y ago
Sentiment analysis is something the Turkers can do quickly and efficiently, so those tasks tend to go lower because you can recruit such a larger worker pool for them. Same for image classifications.
grogenaut 13y ago
good sentiment or survey results generally run more like $.75-$2

linux_devil 13y ago

Its been there for long time, I don't understand why is he glorifying what Google has achieved in ages. Honestly , I don't find anything new in technique.