7 ms·

Generating MIDI melody from lyrics using LSTM-GANs

120 points by groar 7y ago

dimmuborgir 7y ago
Judging by the four provided melodies: 1) the notes have very little rhythmic variation. 2) the melodies don't seem to have any concept of metre or metric accent.
thomasfl 7y ago
Oh no, they're ruining popular music as we know it. Anybody can just push some buttons and generate the next hit song. All you need is some with artificial lyrics, artificial melody, artificial vocal (Yamaha vocaloid), on top of a beat bought on the net.
- narrator 7y ago
  People idolizing pop stars is something I would be glad to get rid of. Democratize the idols I say. Everyone gets to be a idol!
  - Juliate 7y ago
    Teaching music practice does wonders in this regard. :)
- Piskvorrr 7y ago
  So the next hit song will be indistinguishable from the others, except for the method of generation? Welp...
- carlbarrdahl 7y ago
  When something hard becomes easy with the aid of technology, our expectations and demands of quality tend to rise. Maybe this is a good thing? What great innovations or paradigms in music has happened recently? For example, what hiphop did in the way it combines beats and poetry revolutionized music as we know it.
- nkrisc 7y ago
  I know you're being sarcastic but I always thought the concern over the provenance of art is strange. If a computer can generate good music, why does it matter that a computer created it if it's good music? And if computers are just spitting out terrible music, who cares? Same goes for computer generated visual art, painting created by elephants, or whatever else you can think of that challenges the position of humans as the sole creators of art.
  - Fnoord 7y ago
    Supposedly because it devalues the value of manual created artwork, or because people are afraid for change. > And if computers are just spitting out terrible music, who cares? Well, I loved the 90s goa trance, but it got snowed under by the 00s psytrance. If that 00s psytrance was never made available, perhaps there would've been more of the style I preferred. My point being that: 1) What is popular matters A) in ways which might not be apparent at face glance. B) even if you specifically dislike it.
  - Juliate 7y ago
    Depends if you consider it pure raw material (I think it is) that you can use to convey something of your own or if you can relate to some point to it (and do the same, but to some further extent). At a rather utilitarian level, one wouldn't consider a spoon exactly the same way whether it was: 1) carved out of wood by someone you can relate to (by history, reasons, knowledge); as a unique piece, or part of a series; 2) or built in a factory, by workers; and available in the same fashion in X instances; 3) or built in a factory by robots; and available as an exact copy in X instances; 4) or a pure unique artefact that got the practical shape of a spoon. And that would be for a spoon only. I am of the school of thought that one doesn't make art without the intent of communicating/transmiting (it/through it) with other people, however remote/intimate it can be. If only because one cannot live alone. In this line, whatever pleasing something can be, if there's no transmission/communication/need/desire behind the creation of it, if there is no social reason behind it, it has no resonance, no affect to me that makes me actually relate to it (and to the person that thought of it, fell on it, carved it in their words, sounds, silences, colors, shapes, steps, etc.). To me, whatever the artist intent was, when anything is done, it is done by someone - this someone cared, if only enough to make it, and that way attached something to their work. That is something I can try to relate to (to the product itself, and to the intent perhaps). When something is produced by a system, there still may be the intent of the person(s) that designed this system. But it may also be totally independant of the designers scope. That is interesting also, as pure rough material. But it isn't remotely as profound, interesting, human as something that is done by someone real.
- archi42 7y ago
  Why the beat from the net? Just analyze the lyric's measure and produce a beat based on that. I know, it's lame because that doesn't even need ML :P Or, as a drummer, I recommend just going with (1)bumm (2)tzz (3)bumm (4)tz :(goto 1), maybe occasionally inserting a (2+)bumm if you're feeling fancy. Benchmark proposal: It's good if a significant number out of N=1000 prefer its output for The Raven (by Poe) over Alan Parsons hand-crafted version. parent seems /s, this is /s.
- fit2rule 7y ago
  Hello, the 80's called, your moaning about the boring synth musicians interests is deleted. We synth nerds have been having precisely this argument for decades now, and the answers always, unequivocally is, no matter how you made the music, its the musician, not the tool... as soon as we all agree "computer music is boring", someone proves us wrong.
  - filoleg 7y ago
    This. People seem to not realize that there is much more complexity to music besides just the melody and the chord progression used. Which sort of makes sense, because those are probably the only things that people not familiar with making music can easily grasp. Did the music of the past 10 years do much in those aspects? No, because chord progressions and melodies have been explored into such depths over centuries, it is kinda difficult to do anything new there without sounding ultra experimental and, most likely, not palpable to an average ear. But the music has made giant leaps in terms of sound synthesis, timbre complexity, etc. Midi generation from lyrics is imo a cool trick, but it doesn't even begin to scratch the surface of "automatic the music-making". It's like saying that using a TypeScript transpiler with syntactic sugar on top is getting us to the point of Javascript code writing itself without need for any software engineers.
    - fit2rule 7y ago
      I could see this being used in the synth-core realm quite comically.
- matchagaucho 7y ago
  That's the nature of "pop" music. Too much artificiality one week will spawn an acoustic counterculture movement the next week.
deleted 7y ago
[deleted]
NPMaxwell 7y ago
Music, like weaving, is a predecessor in using algorithms to do work. For hundreds of years, music theorists have been codifying the algorithms, or creating new algorithms, that create good music (for particular definitions of good). Like other code instantiated in a network, this code is more tailored to specific prior states and is less available to analysis of its details than prior efforts.
gwern 7y ago
> Q5 In the case of Markov, etc., I understand that the generation may be not very conformant to the style learnt, unless using a high order Markov but with the risk of recopying entire sequences from the corpus and thus plagiat. But, in the case of a RNN-based architecture [9], what is the rationale?-------------------- A5: As mentioned before, RNN does the similar work as LSTM in our work. But without including a discriminator, it only learns transmission probability between adjacent notes, but does not promise that generated sequences look like real ones. Come on, guys, that's just not true. You do not need an adversarial loss to get good quality melodies. Look at Sturm's char-RNN on ABC notation, or OpenAI's MuseNet, or a bunch of Project Magenta work, or my own GPT-2 ABC music (MIDI in progress): https://www.gwern.net/GPT-2-music https://www.gwern.net/GPT-2-music Or for that matter, any generative model trained with a non-adversarial loss (anything using GPT-2 for example). In fact, generally, everyone avoids GANs for sequence generation because they work so badly compared to regular likelihood training... (Just at a skim, their 'baseline' is pretty suspicious. I'd expect an ablation for the GAN, not comparing their 400-unit LSTM to... a 100-unit LSTM https://www.aclweb.org/anthology/N19-4015.pdf https://www.aclweb.org/anthology/N19-4015.pdf ? Really?)
- zozbot234 7y ago
  feynmanliang's BachBot and hexahedria's biaxial-rnn also have very good melody generation, and these deal with polyphonic music which is quite a bit harder than simple monody/monophonic music.
tabtab 7y ago
I always wanted a "demo engine" whereby one feeds in the melody and chord name, and then a style(s). The AI would then use pattern matching to make a fuller score in the chosen style(s). The output could be midi and/or an audio file (such as .WAV). Bonus points for vocals if given lyrics. I could make Elvis diet parodies: "Ain't nothing but a round dog..." Band-in-a-Box software sort of does this, but lacks realism in my opinion.

120 points by groar 7y ago

Judging by the four provided melodies: 1) the notes have very little rhythmic variation. 2) the melodies don't seem to have any concept of metre or metric accent.

thomasfl 7y ago

Oh no, they're ruining popular music as we know it. Anybody can just push some buttons and generate the next hit song. All you need is some with artificial lyrics, artificial melody, artificial vocal (Yamaha vocaloid), on top of a beat bought on the net.

narrator 7y ago
People idolizing pop stars is something I would be glad to get rid of. Democratize the idols I say. Everyone gets to be a idol!
- Juliate 7y ago
  Teaching music practice does wonders in this regard. :)
Piskvorrr 7y ago
So the next hit song will be indistinguishable from the others, except for the method of generation? Welp...
carlbarrdahl 7y ago
When something hard becomes easy with the aid of technology, our expectations and demands of quality tend to rise. Maybe this is a good thing? What great innovations or paradigms in music has happened recently? For example, what hiphop did in the way it combines beats and poetry revolutionized music as we know it.
nkrisc 7y ago
I know you're being sarcastic but I always thought the concern over the provenance of art is strange. If a computer can generate good music, why does it matter that a computer created it if it's good music? And if computers are just spitting out terrible music, who cares? Same goes for computer generated visual art, painting created by elephants, or whatever else you can think of that challenges the position of humans as the sole creators of art.
- Fnoord 7y ago
  Supposedly because it devalues the value of manual created artwork, or because people are afraid for change. > And if computers are just spitting out terrible music, who cares? Well, I loved the 90s goa trance, but it got snowed under by the 00s psytrance. If that 00s psytrance was never made available, perhaps there would've been more of the style I preferred. My point being that: 1) What is popular matters A) in ways which might not be apparent at face glance. B) even if you specifically dislike it.
- Juliate 7y ago
  Depends if you consider it pure raw material (I think it is) that you can use to convey something of your own or if you can relate to some point to it (and do the same, but to some further extent). At a rather utilitarian level, one wouldn't consider a spoon exactly the same way whether it was: 1) carved out of wood by someone you can relate to (by history, reasons, knowledge); as a unique piece, or part of a series; 2) or built in a factory, by workers; and available in the same fashion in X instances; 3) or built in a factory by robots; and available as an exact copy in X instances; 4) or a pure unique artefact that got the practical shape of a spoon. And that would be for a spoon only. I am of the school of thought that one doesn't make art without the intent of communicating/transmiting (it/through it) with other people, however remote/intimate it can be. If only because one cannot live alone. In this line, whatever pleasing something can be, if there's no transmission/communication/need/desire behind the creation of it, if there is no social reason behind it, it has no resonance, no affect to me that makes me actually relate to it (and to the person that thought of it, fell on it, carved it in their words, sounds, silences, colors, shapes, steps, etc.). To me, whatever the artist intent was, when anything is done, it is done by someone - this someone cared, if only enough to make it, and that way attached something to their work. That is something I can try to relate to (to the product itself, and to the intent perhaps). When something is produced by a system, there still may be the intent of the person(s) that designed this system. But it may also be totally independant of the designers scope. That is interesting also, as pure rough material. But it isn't remotely as profound, interesting, human as something that is done by someone real.
archi42 7y ago
Why the beat from the net? Just analyze the lyric's measure and produce a beat based on that. I know, it's lame because that doesn't even need ML :P Or, as a drummer, I recommend just going with (1)bumm (2)tzz (3)bumm (4)tz :(goto 1), maybe occasionally inserting a (2+)bumm if you're feeling fancy. Benchmark proposal: It's good if a significant number out of N=1000 prefer its output for The Raven (by Poe) over Alan Parsons hand-crafted version. parent seems /s, this is /s.
fit2rule 7y ago
Hello, the 80's called, your moaning about the boring synth musicians interests is deleted. We synth nerds have been having precisely this argument for decades now, and the answers always, unequivocally is, no matter how you made the music, its the musician, not the tool... as soon as we all agree "computer music is boring", someone proves us wrong.
- filoleg 7y ago
  This. People seem to not realize that there is much more complexity to music besides just the melody and the chord progression used. Which sort of makes sense, because those are probably the only things that people not familiar with making music can easily grasp. Did the music of the past 10 years do much in those aspects? No, because chord progressions and melodies have been explored into such depths over centuries, it is kinda difficult to do anything new there without sounding ultra experimental and, most likely, not palpable to an average ear. But the music has made giant leaps in terms of sound synthesis, timbre complexity, etc. Midi generation from lyrics is imo a cool trick, but it doesn't even begin to scratch the surface of "automatic the music-making". It's like saying that using a TypeScript transpiler with syntactic sugar on top is getting us to the point of Javascript code writing itself without need for any software engineers.
  - fit2rule 7y ago
    I could see this being used in the synth-core realm quite comically.
matchagaucho 7y ago
That's the nature of "pop" music. Too much artificiality one week will spawn an acoustic counterculture movement the next week.

deleted 7y ago

[deleted]

NPMaxwell 7y ago

Music, like weaving, is a predecessor in using algorithms to do work. For hundreds of years, music theorists have been codifying the algorithms, or creating new algorithms, that create good music (for particular definitions of good). Like other code instantiated in a network, this code is more tailored to specific prior states and is less available to analysis of its details than prior efforts.

gwern 7y ago

> Q5 In the case of Markov, etc., I understand that the generation may be not very conformant to the style learnt, unless using a high order Markov but with the risk of recopying entire sequences from the corpus and thus plagiat. But, in the case of a RNN-based architecture [9], what is the rationale?-------------------- A5: As mentioned before, RNN does the similar work as LSTM in our work. But without including a discriminator, it only learns transmission probability between adjacent notes, but does not promise that generated sequences look like real ones. Come on, guys, that's just not true. You do not need an adversarial loss to get good quality melodies. Look at Sturm's char-RNN on ABC notation, or OpenAI's MuseNet, or a bunch of Project Magenta work, or my own GPT-2 ABC music (MIDI in progress): https://www.gwern.net/GPT-2-music https://www.gwern.net/GPT-2-music Or for that matter, any generative model trained with a non-adversarial loss (anything using GPT-2 for example). In fact, generally, everyone avoids GANs for sequence generation because they work so badly compared to regular likelihood training... (Just at a skim, their 'baseline' is pretty suspicious. I'd expect an ablation for the GAN, not comparing their 400-unit LSTM to... a 100-unit LSTM https://www.aclweb.org/anthology/N19-4015.pdf https://www.aclweb.org/anthology/N19-4015.pdf ? Really?)

zozbot234 7y ago
feynmanliang's BachBot and hexahedria's biaxial-rnn also have very good melody generation, and these deal with polyphonic music which is quite a bit harder than simple monody/monophonic music.

tabtab 7y ago

I always wanted a "demo engine" whereby one feeds in the melody and chord name, and then a style(s). The AI would then use pattern matching to make a fuller score in the chosen style(s). The output could be midi and/or an audio file (such as .WAV). Bonus points for vocals if given lyrics. I could make Elvis diet parodies: "Ain't nothing but a round dog..." Band-in-a-Box software sort of does this, but lacks realism in my opinion.