4 ms·

Hey, I just installed and played around with reactionRNN; I hate to be negative (especially to people open sourcing models, kudos!), but your model seems to per

by rm999 9y ago

Hey, I just installed and played around with reactionRNN; I hate to be negative (especially to people open sourcing models, kudos!), but your model seems to perform quite poorly. It immediately failed my "easy" smell tests: https://i.imgur.com/FvfuZgy.png https://i.imgur.com/FvfuZgy.png, and didn't really work on most of my other tests: "This book sucks" is 0% angry, "I'm going to go home and listen to emo music and cry" is 0% sad, "Check out this hilarious youtube video" is only 26% haha. Your example "He was only 41." is 100% sad, but "He was only 42." is 0% sad. These aren't hand-picked, these are literally things I just typed in. From what I can tell it usually gets anything negative wrong, and usually picks "haha" for the positive ones. Subjective performance is worse than your included examples. I've been building models my whole career, and what I've learned is most people will take claimed performance at face value until it burns them. It's beneficial to no one if someone comes up with an idea based off your repo description, builds it out, then finds it doesn't work adequately. My advice is to update your examples and test cases, and keep finding ways to improve the model.

minimaxir 9y ago
The result is the reaction to a given text; it won't fully be the same as a sentiment analysis, unfortunately. Additionally, as I put in the README notes, keep in mind that the network is trained on modern (2016-2017) language. As a result, inputting rhetorical/ironic statements will often yield love/wow responses and not sad/angry. That type of systemic bias in text analysis is currently unsolved and there isn't an easy way to account for it. "I am so angry" and "I'm going to go home and listen to emo music and cry" are phrases that would likely be posted on Facebook ironically, and therefore classifying it tricky. For the other examples, yes, that result might be overfitting on characters and while setting up the model I had difficulty accounting for that while still getting the model to converge. I'll admit it's not a perfect model (it was a side project while I was frustrated during a job hunt), but it's a great proof of concept. Unfortunately, since Facebook crippled their /posts endpoint, I can't get more data to improve the model...

by rm999 9y ago

minimaxir 9y ago

The result is the reaction to a given text; it won't fully be the same as a sentiment analysis, unfortunately. Additionally, as I put in the README notes, keep in mind that the network is trained on modern (2016-2017) language. As a result, inputting rhetorical/ironic statements will often yield love/wow responses and not sad/angry. That type of systemic bias in text analysis is currently unsolved and there isn't an easy way to account for it. "I am so angry" and "I'm going to go home and listen to emo music and cry" are phrases that would likely be posted on Facebook ironically, and therefore classifying it tricky. For the other examples, yes, that result might be overfitting on characters and while setting up the model I had difficulty accounting for that while still getting the model to converge. I'll admit it's not a perfect model (it was a side project while I was frustrated during a job hunt), but it's a great proof of concept. Unfortunately, since Facebook crippled their /posts endpoint, I can't get more data to improve the model...