3 ms·

Reminds me of the stat I heard that humans are only 70% accurate at sentiment analysis, because different people will not agree on the appropriate sentiment lab

Reminds me of the stat I heard that humans are only 70% accurate at sentiment analysis, because different people will not agree on the appropriate sentiment label. That sets a theoretical limit on the effectiveness of machine-learning algorithms, because if humans can't agree, then any product that needs to take an opinion is going to be wrong 30% of the time. (This is probably also why Big Tech companies are leaning so heavily into personalization.) Also reminds me of when I asked a veteran therapist what the most surprising part of his job was, and it was: 1.) The variety of ways that different people perceive a given situation, and just how much neurodiversity is out there. 2.) How everybody expects that everyone else will see the situation exactly the same way they do.

wongarsu 4y ago
That sets the theoretical limit for an algorithm trained on a dataset labeled by outsiders. Most people should be able to label the sentiment of their own statements with much higher accuracy. Making such a dataset is much harder than letting Mechanical Turk workers label reddit comments, and you somehow have to set up a situation where people are honest about their labels, but the rewards might be worth it.
- nostrademons 4y ago
  That's the personalization angle. A lot of effort's being expended on transfer learning + training at the edge, where you start with a general model trained on humanity and then it gradually learns about the specific human(s) it's interacting with.
  - wongarsu 4y ago
    My idea is more along the lines of asking each redditor to label the sentiment of 20 of their own (recent) comments, building a dataset and model from that, instead of having unrelated people guess what they meant. Personalizing to the actual subculture the interaction takes place in would be another step. You probably need both.
kortex 4y ago
You definitely need more labels when categorizing sentiment than just emotion/valence. Possibly even degrees/confidence by the annotator. Heck, have several annotators label it, and derive a "Controversial" feature based on the spread of the ratings. Only tangentially related, is your username a pun on "Nostradamus", "Nostril", and "Nasal Demons"? If so, that is very witty! http://www.catb.org/jargon/html/N/nasal-demons.html http://www.catb.org/jargon/html/N/nasal-demons.html

wongarsu 4y ago

That sets the theoretical limit for an algorithm trained on a dataset labeled by outsiders. Most people should be able to label the sentiment of their own statements with much higher accuracy. Making such a dataset is much harder than letting Mechanical Turk workers label reddit comments, and you somehow have to set up a situation where people are honest about their labels, but the rewards might be worth it.

nostrademons 4y ago
That's the personalization angle. A lot of effort's being expended on transfer learning + training at the edge, where you start with a general model trained on humanity and then it gradually learns about the specific human(s) it's interacting with.
- wongarsu 4y ago
  My idea is more along the lines of asking each redditor to label the sentiment of 20 of their own (recent) comments, building a dataset and model from that, instead of having unrelated people guess what they meant. Personalizing to the actual subculture the interaction takes place in would be another step. You probably need both.

kortex 4y ago

You definitely need more labels when categorizing sentiment than just emotion/valence. Possibly even degrees/confidence by the annotator. Heck, have several annotators label it, and derive a "Controversial" feature based on the spread of the ratings. Only tangentially related, is your username a pun on "Nostradamus", "Nostril", and "Nasal Demons"? If so, that is very witty! http://www.catb.org/jargon/html/N/nasal-demons.html http://www.catb.org/jargon/html/N/nasal-demons.html