4 ms·

I think you might have a misunderstanding. The "error" was misclassifying the sentiment of this imdb review [1], the human labeled it as positive but the LLM la

I think you might have a misunderstanding. The "error" was misclassifying the sentiment of this imdb review [1], the human labeled it as positive but the LLM labeled it as negative. The researchers concluded that the model was more sensitive to sarcasm than the human reviewer. https://www.imdb.com/review/rw1756778/ https://www.imdb.com/review/rw1756778/

deleted 4y ago
[deleted]
m3kw9 4y ago
It’s a poorly written review and anyone reading wouldn’t find it useful. Note that 1/6 found it useful in the site
- ajross 4y ago
  It's not that poorly written. But regardless: it's clearly a poorly written, sarcastic, negative review. Read it a second time and you'll pick out the sarcasm for sure. The reviewer thought this was a dumb film. The first notable point is that the model caught that (on its first read, FWIW), even though the original human doing the labeling didn't. And the second is that the researchers discovered this, and presumably discussed it. And yet when they wrote up the paper they not only dropped the content but denied that the analysis had been done.
  - gleenn 4y ago
    Can you please specify at least a part or two that you specifically feel are sarcastic? The author doesn't seem to like the movie but I feel like the majority of the comments are quite factual or seem like straightforward opinion.
    - knaik94 4y ago
      The part of the review where it says "he must have never in his life seen a flick about any small towns" could be classified as sarcastic. The reviewer also says "you should watch the movie" if you are curious about the ending, but the overall intent of the review seems to be the opposite.
      - hn_go_brrrrr 4y ago
        I would call that acerbic. There's none of the reversal in sense I expect to see in sarcasm.
        sdenton4 4y ago
        in 2023 the goalposts on agi have moved far enough that we are in the bleachers arguing about whether a model was correct to classify a movie review as sarcastic, when it may have merely been acerbic.
    - deadpannini 4y ago
      The opening qualifies as sarcastic: > I am so happy not to live in an American small town. Because whenever I'm shown some small town in the States it is populated with all kinds of monsters among whom flesh hungry zombies, evil aliens and sinister ghosts are most harmless. Mocking irony, in the context of a negative review.
      - gleenn 4y ago
        Sarcasm is saying one thing and meaning the opposite. If monsters are most harmless, then also not wanting to live there makes logical sense and isn't backwards. So that's straightforward mockery, not sarcastic.
        ajross 4y ago
        > Sarcasm is saying one thing and meaning the opposite. What I find fascinating here is how generative AI has inverted all our sci-fi tropes. I mean, sure: you're right! That's the way "sarcasm" is defined in most dictionaries. But you and I both know that as the language is actually used, the term means a whole host of techniques used to convey negative emotional content in language that is not directly negative. Your (correct!) dictionary pedantry isn't interesting to me. We've been here before. But GPT-4 wasn't trained on dictionary rules. It was trained on actual language. And it's actually better at inferring this stuff than the pedants are. Our introvert brains have trouble teasing meaning like this and have to hide behind rules and structure. The computer doesn't. To wit: the robot is us.
        gleenn 4y ago
        But I'm not being a pedant, we're measuring sarcasm. Someone has to clearly define it to judge the AI. If everyone in these threads thinks sarcasm is anything "negative" then they are wrong. That would be a negative sentiment classification. You have to be clear with what you're measuring.
  - thaumasiotes 4y ago
    It's not clear to me that it's a negative review. The movie is rated as "artistically worse than a movie by Oliver Stone". Other than that, the message appears to be that the movie is typical of its genre, which is not generally considered a bad thing. There is another message that the reviewer doesn't like the genre, but that isn't a comment on the movie.
jay_kyburz 4y ago
The last sentence, the conclusion, says it's like some other move, but worse. Way worse. There is no way a human reading this review would think its positive. "I'll just tell you that it's slightly reminiscent of U Turn by Oliver Stone but is a way down in all artistic properties." Oh and UTurn only scores 6.7 so its a bit of a crap movie to start with.

deleted 4y ago

[deleted]

m3kw9 4y ago

It’s a poorly written review and anyone reading wouldn’t find it useful. Note that 1/6 found it useful in the site

ajross 4y ago
It's not that poorly written. But regardless: it's clearly a poorly written, sarcastic, negative review. Read it a second time and you'll pick out the sarcasm for sure. The reviewer thought this was a dumb film. The first notable point is that the model caught that (on its first read, FWIW), even though the original human doing the labeling didn't. And the second is that the researchers discovered this, and presumably discussed it. And yet when they wrote up the paper they not only dropped the content but denied that the analysis had been done.
- gleenn 4y ago
  Can you please specify at least a part or two that you specifically feel are sarcastic? The author doesn't seem to like the movie but I feel like the majority of the comments are quite factual or seem like straightforward opinion.
  - knaik94 4y ago
    The part of the review where it says "he must have never in his life seen a flick about any small towns" could be classified as sarcastic. The reviewer also says "you should watch the movie" if you are curious about the ending, but the overall intent of the review seems to be the opposite.
    - hn_go_brrrrr 4y ago
      I would call that acerbic. There's none of the reversal in sense I expect to see in sarcasm.
      - sdenton4 4y ago
        in 2023 the goalposts on agi have moved far enough that we are in the bleachers arguing about whether a model was correct to classify a movie review as sarcastic, when it may have merely been acerbic.
  - deadpannini 4y ago
    The opening qualifies as sarcastic: > I am so happy not to live in an American small town. Because whenever I'm shown some small town in the States it is populated with all kinds of monsters among whom flesh hungry zombies, evil aliens and sinister ghosts are most harmless. Mocking irony, in the context of a negative review.
    - gleenn 4y ago
      Sarcasm is saying one thing and meaning the opposite. If monsters are most harmless, then also not wanting to live there makes logical sense and isn't backwards. So that's straightforward mockery, not sarcastic.
      - ajross 4y ago
        > Sarcasm is saying one thing and meaning the opposite. What I find fascinating here is how generative AI has inverted all our sci-fi tropes. I mean, sure: you're right! That's the way "sarcasm" is defined in most dictionaries. But you and I both know that as the language is actually used, the term means a whole host of techniques used to convey negative emotional content in language that is not directly negative. Your (correct!) dictionary pedantry isn't interesting to me. We've been here before. But GPT-4 wasn't trained on dictionary rules. It was trained on actual language. And it's actually better at inferring this stuff than the pedants are. Our introvert brains have trouble teasing meaning like this and have to hide behind rules and structure. The computer doesn't. To wit: the robot is us.
        gleenn 4y ago
        But I'm not being a pedant, we're measuring sarcasm. Someone has to clearly define it to judge the AI. If everyone in these threads thinks sarcasm is anything "negative" then they are wrong. That would be a negative sentiment classification. You have to be clear with what you're measuring.
- thaumasiotes 4y ago
  It's not clear to me that it's a negative review. The movie is rated as "artistically worse than a movie by Oliver Stone". Other than that, the message appears to be that the movie is typical of its genre, which is not generally considered a bad thing. There is another message that the reviewer doesn't like the genre, but that isn't a comment on the movie.

jay_kyburz 4y ago

The last sentence, the conclusion, says it's like some other move, but worse. Way worse. There is no way a human reading this review would think its positive. "I'll just tell you that it's slightly reminiscent of U Turn by Oliver Stone but is a way down in all artistic properties." Oh and UTurn only scores 6.7 so its a bit of a crap movie to start with.