6 ms·

So presumably the purpose of this experiment was to quantify just how bad algorithms can be at inferring meaning? Trying to measure the sadness of lyrics with

by 3chelon 10y ago

So presumably the purpose of this experiment was to quantify just how bad algorithms can be at inferring meaning? Trying to measure the sadness of lyrics with an algo is tantamount to the early scene in "Dead Poets' Society" where the students are supposed to measure the greatness of poetry by plotting it on a chart. It's demonstrated in the data: one of the most depressing songs of their entire oeuvre is "Fitter, Happier" but because it's completely metaphorical (or sarcastic), the algo rates it as one of the happiest. This will always be a big problem in AI. Measuring the literal meaning of words is easy. Measuring the semantics and context of words is a completely different matter.

jmmcd 10y ago
> This will always be a big problem in AI I agreed with everything you wrote except for "always".
- Grue3 10y ago
  Considering even actual humans often have disagreements about meanings of songs, I'm not sure how an AI is supposed to deduce the "correct" one.
  - andrepd 10y ago
    Neither am I. But 10 years ago I would have said I had no idea how an AI was supposed to recognise objects in a picture, and now they do it better than humans. Tech moves fast.
    - jmmcd 10y ago
      Recognising objects in pictures is something that, in principle, a machine could do better than humans, because there is ground truth. I think it's not possible in principle for a machine to do better than humans at recognising human emotions, because a human can't exactly be wrong at this. (Other than edge cases such as not knowing the right name for an emotion, perhaps.) But suppose a sample of 1000 humans answered a survey on emotional content of a song and a machine was able to predict the distribution within some tolerance. In a sense, this would be superhuman performance, but even if we don't like that interpretation, it would still be very good performance. And it is possible in principle. So the argument that humans sometimes disagree doesn't prevent a machine doing well on the task.
      - anewhnaccount 10y ago
        Another possible output for this type of task is an opinion or interpretation and a justification of it. A machine might feasibly be considered "good" if it output an unpopular interpretation with a good justification.
        jmmcd 10y ago
        Great point! The same type of argument is going in the AI subfield of computational creativity. A computational system might produce a piece (of art or music, say) which nobody initially likes, but could provide a justification/interpretation which might be convincing. In some ways this would be a big improvement over a pure Turing test-style judgement of the output itself. I think Margaret Boden has written this idea into her definition of computational creativity. EDIT: "The ultimate vindication of AI-creativity would be a program that generated novel ideas which initially perplexed or even repelled us, but which was able to persuade us that they were indeed valuable.", from Boden, Creativity and Artificial Intelligence, 1998. I believe, based on her other writings, that she thinks of it as a sufficient, but not a necessary condition, for AI creativity.
  - devishard 10y ago
    I don't think that the imprecision of human song interpretation can be used as evidence that AIs are inaccurate.
- 3chelon 10y ago
  I thought before writing "always". But I do think it will be difficult for the foreseeable future, because as @Grue3 already pointed out, it's ambiguous even to humans. The central problem of getting a machine to understand metaphor, irony and the like is going to be Hard by any definition.
  - jmmcd 10y ago
    Being ambiguous to humans doesn't make it an ill-defined task. A machine which perfectly mimicked the judgements of just one human would be doing pretty well, well enough that we couldn't justify saying "judging emotion is a big problem for AI" unless we said the same thing about humans. Having said that, I agree that it seems a hard problem for the foreseeable future.
HenryTheHorse 10y ago
> So presumably the purpose of this experiment was to quantify just how bad algorithms can be at inferring meaning? That's a bit harsh, but I agree. Words & music are both key to understanding a song's "mood". Only looking at words can completely strip the irony/sarcasm off a song (as can only paying attention to the arrangements and melody/harmony.) How would such analysis even work on artists like Dylan?
- beanmunch 10y ago
  They use Spotify's valence score as a measure of the mood of the music.
tenderlove 10y ago
I think the purpose of this experiment was to have some fun.
- 3chelon 10y ago
  Indeed :)

by 3chelon 10y ago

jmmcd 10y ago

> This will always be a big problem in AI I agreed with everything you wrote except for "always".

Grue3 10y ago
Considering even actual humans often have disagreements about meanings of songs, I'm not sure how an AI is supposed to deduce the "correct" one.
- andrepd 10y ago
  Neither am I. But 10 years ago I would have said I had no idea how an AI was supposed to recognise objects in a picture, and now they do it better than humans. Tech moves fast.
  - jmmcd 10y ago
    Recognising objects in pictures is something that, in principle, a machine could do better than humans, because there is ground truth. I think it's not possible in principle for a machine to do better than humans at recognising human emotions, because a human can't exactly be wrong at this. (Other than edge cases such as not knowing the right name for an emotion, perhaps.) But suppose a sample of 1000 humans answered a survey on emotional content of a song and a machine was able to predict the distribution within some tolerance. In a sense, this would be superhuman performance, but even if we don't like that interpretation, it would still be very good performance. And it is possible in principle. So the argument that humans sometimes disagree doesn't prevent a machine doing well on the task.
    - anewhnaccount 10y ago
      Another possible output for this type of task is an opinion or interpretation and a justification of it. A machine might feasibly be considered "good" if it output an unpopular interpretation with a good justification.
      - jmmcd 10y ago
        Great point! The same type of argument is going in the AI subfield of computational creativity. A computational system might produce a piece (of art or music, say) which nobody initially likes, but could provide a justification/interpretation which might be convincing. In some ways this would be a big improvement over a pure Turing test-style judgement of the output itself. I think Margaret Boden has written this idea into her definition of computational creativity. EDIT: "The ultimate vindication of AI-creativity would be a program that generated novel ideas which initially perplexed or even repelled us, but which was able to persuade us that they were indeed valuable.", from Boden, Creativity and Artificial Intelligence, 1998. I believe, based on her other writings, that she thinks of it as a sufficient, but not a necessary condition, for AI creativity.
- devishard 10y ago
  I don't think that the imprecision of human song interpretation can be used as evidence that AIs are inaccurate.
3chelon 10y ago
I thought before writing "always". But I do think it will be difficult for the foreseeable future, because as @Grue3 already pointed out, it's ambiguous even to humans. The central problem of getting a machine to understand metaphor, irony and the like is going to be Hard by any definition.
- jmmcd 10y ago
  Being ambiguous to humans doesn't make it an ill-defined task. A machine which perfectly mimicked the judgements of just one human would be doing pretty well, well enough that we couldn't justify saying "judging emotion is a big problem for AI" unless we said the same thing about humans. Having said that, I agree that it seems a hard problem for the foreseeable future.

HenryTheHorse 10y ago

> So presumably the purpose of this experiment was to quantify just how bad algorithms can be at inferring meaning? That's a bit harsh, but I agree. Words & music are both key to understanding a song's "mood". Only looking at words can completely strip the irony/sarcasm off a song (as can only paying attention to the arrangements and melody/harmony.) How would such analysis even work on artists like Dylan?

beanmunch 10y ago
They use Spotify's valence score as a measure of the mood of the music.

tenderlove 10y ago

I think the purpose of this experiment was to have some fun.

3chelon 10y ago
Indeed :)