8 ms·
My main take away here is that Turkers are terrible at some of these tasks. The "stance" task is, "Classify the tweet as having a positive stance towards Sectio
by Imnimo 4y ago
My main take away here is that Turkers are terrible at some of these tasks. The "stance" task is, "Classify the tweet as having a positive stance towards Section 230, a negative stance, or a neutral stance.", and the Turkers accuracy was like 20%.
Even in its best task, ChatGPT only got 75% accuracy.
- p1necone 4y ago20% means you can beat ChatGPT if you invert the answers!
- quickthrower2 4y agoThere are 3 stances. But 33% is achievable by chance.
- xyzzyz 4y agoOnly if probability distribution of answers is known.
- riku_iki 4y agojust speculating, there is a chance ChatGPT was trained on this dataset in the past, and learned probability distribution..
- raincole 4y agoEh, even if the probability distribution is unknown, a random guess should still have 33% chance to be correct.
- p1necone 4y agoI started explaining why you were wrong and got about 5 words in before I realized you're correct.
- quickthrower2 4y agoProbability: always counterintuitive
- raincole 4y agoIt's just how you frame the question (in this parcitular case). If random guess were not 33% correct, you effectively found a way to "win" in rock-paper-scissor.
- dotancohen 4y agoI do have a strategy for always winning rock-paper-scissors. Play often for unimportant reasons, and always choose "rock". Now that your adversaries are conditioned to expect you to choose "rock", when an important decision is being decided choose "scissors".
- p1necone 4y agoI wonder if there are any reliable techniques for generating good random/pseudorandom numbers mentally. I guess you could just run the same algorithm computers do but it would probably be kinda taxing. I wonder if a system that uses some arbitrary features of what you can currently see in front of you would be unpredictable enough?
- deleted 4y ago[deleted]
- emodendroket 4y agoA cynic might say that someone who could do that task well has better things to do with their time.
- CGamesPlay 4y agoI have long suspected that Turkers dishonestly perform the tasks. At 0.06 cents per task, you're really incentivizing "finish the task as quickly as possible"; and "press the left button" is a lot faster than "read the tweet, think about it, and classify".
- ftxbro 4y ago> At 0.06 cents per task Do you work at Verizon?
- CGamesPlay 4y agoI got that figure from https://news.ycombinator.com/item?id=35335558 https://news.ycombinator.com/item?id=35335558 [append] Oh, it's the xkcd Verizon agent reference. Got it. My bad.
- ftxbro 4y agoI was nitpicking about 0.06 dollars versus 0.06 cents. It's this ancient meme, where a guy records a customer service call with Verizon where they have some back-and-forth about whether these are the same or not. https://www.youtube.com/watch?v=MShv_74FNWU https://www.youtube.com/watch?v=MShv_74FNWU
- renewiltord 4y agoYou always use multiple MTurks on same job when you use them.
- CGamesPlay 4y agoOK, but if all MTurks are doing this (because why wouldn't they) then you are just sampling a random variable.
- johnthewise 4y agoYou are not just taking the average, answers would have to be consistent with each other.
- cormacrelf 4y agoThat’s in the nature of the question, which is probably not representative of what people might want a sentiment analysis for. Section 230 has been the subject of what can only be described as a massive disinformation campaign. Most of the participants were not aware. But ask anybody today what the law even means and you will get 99% completely wrong answers. So even neutral factual statements will be misinterpreted if people think Section 230 is some nebulous but definitely bad thing and so anything about it is probably bad.