4 ms·

Wait so they tested if it detects sarcasm, it failed, and then they were like "let's pretend this never happened" and wrote "We did not test for the ability to

Wait so they tested if it detects sarcasm, it failed, and then they were like "let's pretend this never happened" and wrote "We did not test for the ability to understand sarcasm"?

famouswaffles 4y ago
It didn't fail ?
salad-tycoon 4y ago
There was a conflict of opinions with D3 and human annotator. The quote above at least does not indicate who was right or wrong, merely noting that the machine had a more sensitive sarcasm detector. So I guess maybe it’s saying the human was wrong after all?
- ChatGTP 4y ago
  If you read the IMDB review, there is no sarcasm in it. Maybe more like appreciation with some humour thrown in? Are we all a bit confused and second guessing our own use and interpretation of language now ?
  - knaik94 4y ago
    I got confused about whether the review was positive or negative too. The part of the review where it says "he must have never in his life seen a flick about any small towns" could be classified as sarcastic. The review isn't very clear in terms of prose and conclusion. The reviewer clearly says "you should watch the movie", but after re-reading it a few times, I would consider that as sarcasm too. I can't say for sure whether the reviewer enjoyed it themselves.
    - ChatGTP 4y ago
      Sorry I wasn't clear, I mean appreciation that he does not living in a horrible small town with monsters etc. Not exactly for the movie. "he must have never in his life seen a flick about any small towns" That's probably the least sarcastic sentence for me, because it's just a reinforcement of the opening statement: "I am so happy not to live in an American small town. Because whenever I'm shown some small town in the States it is populated with all kinds of monsters among whom flesh hungry zombies, evil aliens and sinister ghosts are most harmless." I don't think it's a "positive review" I think it's neither, it's fairly neutral, and the author kind of suggests the reader watches the movie. Out of interest I asked someone who has no idea about ChatGPT-4 and it's apparent sarcasm detection abilities and they didn't think it was sarcastic, albeit a bit 'weird' and poorly written. Confirmation bias? We could say more, however, there are more important things to do...
  - dougmwne 4y ago
    It sounded jaded and a little sarcastic to me. I don’t think the reviewer liked the movie much, found it all that original or entertaining.
    - ChatGTP 4y ago
      Then it’s quite subjective then , who’s right ? He gave it a score of 7 which IMO is pretty high?
      - knaik94 4y ago
        Without the context of how the reviewer rates other movies, it's not possible to say whether 7 is high or low. I would say 7 is mid or mediocre, something I wouldn't go out of my way to watch. The trend I've noticed is that most people only use the top half of the scale, anything 5 and less is bad. The funny thing, and I am guilty of this myself, is when they only use 6 through 10 but then use decimal points too.
      - dougmwne 4y ago
        I doubt the score would have been part of sentiment analysis. I ignored it and tried to make a judgement based on the text alone. It seemed more like a 5/10 review to me and mildly negative. And yes it absolutely is subjective. That’s exactly the point and the power of these LLMs, to be able to handle the vagueness of human communication.
knaik94 4y ago
I think you might have a misunderstanding. The "error" was misclassifying the sentiment of this imdb review [1], the human labeled it as positive but the LLM labeled it as negative. The researchers concluded that the model was more sensitive to sarcasm than the human reviewer. https://www.imdb.com/review/rw1756778/ https://www.imdb.com/review/rw1756778/
- deleted 4y ago
  [deleted]
- m3kw9 4y ago
  It’s a poorly written review and anyone reading wouldn’t find it useful. Note that 1/6 found it useful in the site
  - ajross 4y ago
    It's not that poorly written. But regardless: it's clearly a poorly written, sarcastic, negative review. Read it a second time and you'll pick out the sarcasm for sure. The reviewer thought this was a dumb film. The first notable point is that the model caught that (on its first read, FWIW), even though the original human doing the labeling didn't. And the second is that the researchers discovered this, and presumably discussed it. And yet when they wrote up the paper they not only dropped the content but denied that the analysis had been done.
    - gleenn 4y ago
      Can you please specify at least a part or two that you specifically feel are sarcastic? The author doesn't seem to like the movie but I feel like the majority of the comments are quite factual or seem like straightforward opinion.
      - knaik94 4y ago
        The part of the review where it says "he must have never in his life seen a flick about any small towns" could be classified as sarcastic. The reviewer also says "you should watch the movie" if you are curious about the ending, but the overall intent of the review seems to be the opposite.

Wait so they tested if it detects sarcasm, it failed, and then they were like "let's pretend this never happened" and wrote "We did not test for the ability to understand sarcasm"?

famouswaffles 4y ago

It didn't fail ?

salad-tycoon 4y ago

There was a conflict of opinions with D3 and human annotator. The quote above at least does not indicate who was right or wrong, merely noting that the machine had a more sensitive sarcasm detector. So I guess maybe it’s saying the human was wrong after all?

ChatGTP 4y ago
If you read the IMDB review, there is no sarcasm in it. Maybe more like appreciation with some humour thrown in? Are we all a bit confused and second guessing our own use and interpretation of language now ?
- knaik94 4y ago
  I got confused about whether the review was positive or negative too. The part of the review where it says "he must have never in his life seen a flick about any small towns" could be classified as sarcastic. The review isn't very clear in terms of prose and conclusion. The reviewer clearly says "you should watch the movie", but after re-reading it a few times, I would consider that as sarcasm too. I can't say for sure whether the reviewer enjoyed it themselves.
  - ChatGTP 4y ago
    Sorry I wasn't clear, I mean appreciation that he does not living in a horrible small town with monsters etc. Not exactly for the movie. "he must have never in his life seen a flick about any small towns" That's probably the least sarcastic sentence for me, because it's just a reinforcement of the opening statement: "I am so happy not to live in an American small town. Because whenever I'm shown some small town in the States it is populated with all kinds of monsters among whom flesh hungry zombies, evil aliens and sinister ghosts are most harmless." I don't think it's a "positive review" I think it's neither, it's fairly neutral, and the author kind of suggests the reader watches the movie. Out of interest I asked someone who has no idea about ChatGPT-4 and it's apparent sarcasm detection abilities and they didn't think it was sarcastic, albeit a bit 'weird' and poorly written. Confirmation bias? We could say more, however, there are more important things to do...
- dougmwne 4y ago
  It sounded jaded and a little sarcastic to me. I don’t think the reviewer liked the movie much, found it all that original or entertaining.
  - ChatGTP 4y ago
    Then it’s quite subjective then , who’s right ? He gave it a score of 7 which IMO is pretty high?
    - knaik94 4y ago
      Without the context of how the reviewer rates other movies, it's not possible to say whether 7 is high or low. I would say 7 is mid or mediocre, something I wouldn't go out of my way to watch. The trend I've noticed is that most people only use the top half of the scale, anything 5 and less is bad. The funny thing, and I am guilty of this myself, is when they only use 6 through 10 but then use decimal points too.
    - dougmwne 4y ago
      I doubt the score would have been part of sentiment analysis. I ignored it and tried to make a judgement based on the text alone. It seemed more like a 5/10 review to me and mildly negative. And yes it absolutely is subjective. That’s exactly the point and the power of these LLMs, to be able to handle the vagueness of human communication.

knaik94 4y ago

I think you might have a misunderstanding. The "error" was misclassifying the sentiment of this imdb review [1], the human labeled it as positive but the LLM labeled it as negative. The researchers concluded that the model was more sensitive to sarcasm than the human reviewer. https://www.imdb.com/review/rw1756778/ https://www.imdb.com/review/rw1756778/

deleted 4y ago
[deleted]
m3kw9 4y ago
It’s a poorly written review and anyone reading wouldn’t find it useful. Note that 1/6 found it useful in the site
- ajross 4y ago
  It's not that poorly written. But regardless: it's clearly a poorly written, sarcastic, negative review. Read it a second time and you'll pick out the sarcasm for sure. The reviewer thought this was a dumb film. The first notable point is that the model caught that (on its first read, FWIW), even though the original human doing the labeling didn't. And the second is that the researchers discovered this, and presumably discussed it. And yet when they wrote up the paper they not only dropped the content but denied that the analysis had been done.
  - gleenn 4y ago
    Can you please specify at least a part or two that you specifically feel are sarcastic? The author doesn't seem to like the movie but I feel like the majority of the comments are quite factual or seem like straightforward opinion.
    - knaik94 4y ago
      The part of the review where it says "he must have never in his life seen a flick about any small towns" could be classified as sarcastic. The reviewer also says "you should watch the movie" if you are curious about the ending, but the overall intent of the review seems to be the opposite.