6 ms·

Generating MIDI Music with GPT-2

143 points by gwern 7y ago

mebr 7y ago
The author mentions about 5% of the generated music is any good to take a listen. Even the best sounding ones shared don't sound that good to me.
- nr2x 7y ago
  Went in with high hopes, but it’s not remotely listenable.
- narag 7y ago
  Most of it sounds terrible. I guess that it's because there is no way to give some feedback. A chess program knows what means winning. And IIRC "this person doesn't exist" uses human opinion to train. Music has many rules, not only theoretical but unwritten rules about what works. You must incorporate them somehow into the program, either by code or offering something to the program to deduce them.
  - deleted 7y ago
    [deleted]
  - gwern 7y ago
    I previously tried an approach which uses DRL for feedback but I couldn't quite get it to work: https://www.gwern.net/GPT-2-preference-learning https://www.gwern.net/GPT-2-preference-learning At the moment, it probably would be more practical to train a model to predict ratings and use that to screen generated samples or possible completions and throw out too-low-scoring ones (the 'ranker' approach worked out very well for the Meena chatbot recently: https://arxiv.org/abs/2001.09977 https://arxiv.org/abs/2001.09977 )
    - p1esk 7y ago
      What do you think about training a binary classifier to distinguish between human and generated samples? E.g. choose a state of the art model designed to classify composers or styles, and finetune it for this task.
      - mattkrause 7y ago
        That's a tiny step away from a GAN, which works very well for image generation, so seems promising here too.
        p1esk 7y ago
        GANs have been tried many times already for music generation, without much success. GPT-2 works very well for text generation so it seemed promising here too. Music falls somewhere in between text (as a sequence of chords or PCM samples) and image (as a piano roll or a spectrogram), so maybe some hybrid of image and text generators is needed.
        TheOtherHobbes 7y ago
        Music does neither, which is why these naive approaches don't work.
      - gwern 7y ago
        I think that could work potentially as a rejection sampling, but you also have the risk that it will simply find some small discriminative detail and be unusefully good at the classification; that's why you do it in a loop as a GAN, but as you mention, GANs work really badly on sequences, still, so... If you wanted to improve my ABC-MIDI GPT-2, the most straightforward ways would be to do data cleaning (I'm sure there's tens of thousands of awful MIDI files which should be removed! data cleaning with RNNs or GPT-2 or GANs always makes a large difference) and increase the model size (the fact that loss bottomed at 0.20, which is still quite bad, suggests that MIDI is hard enough that GPT-2 is struggling). More interesting would be to use Reformer or another long-range Transformer and try to operate directly on a more raw representation, like the the piano roll representation of MIDI. I think GPT-2 makes a lot of syntax errors which cripple outputs when a 'voice' goes silent, and a piano roll representation would be a lot more robust (at the cost of being like 10x larger).
  - deleted 7y ago
    [deleted]
  - gwern 7y ago
    > And IIRC "this person doesn't exist" uses human opinion to train. It doesn't, incidentally. It's just a standard StyleGAN dumping random images, trained to to model the average image/distribution, and not optimizing for human ratings or anything. Almost all the 'X Does Not Exist' things operate that way, including my own https://www.thiswaifudoesnotexist.net/ https://www.thiswaifudoesnotexist.net/
- meatsock 7y ago
  the rate among humans who are learning to compose is lower as ideas you don't end up using are silent.
- sq_ 7y ago
  I thought it was interesting, even if the overall state of most of them wasn't great. Some of them had decent sections, but then there would be some periodically repeated jarring note. Definitely better than I would've expected.
- themodelplumber 7y ago
  That aspect reminds me of previous experiences with fractal art. If you're evaluating generated sets, there's always going to be a part of an original piece that's off, that's for sure. "Hey, it looks like headlights of cars driving around in the fog! But this other part makes zero sense and is distracting." If you can use a piece as inspiration, browsing around until you find the 5% or whatever % that you like, there's really some benefit to be found. I felt a bit like Simon Cowell, auditioning works and being really picky, last time I did this. But in the end you can discard an item, go with it as-is, adapt it somehow, or use it as reference material. Eventually you build a gallery.
- lostcolony 7y ago
  Yeah...I'm now mildly curious what could be done using this, plus a genetic algorithm decided either by a human listener, or a suitability function encoding basic ideas in music.
- randyrand 7y ago
  "LMD: catchy ambient-esque piano piece" is pretty good. I could see it as background music in a game. I like "Pop MIDI, rapid jazz piano?" too.
  - kybernetikos 7y ago
    You're right. I listened to a few and didn't like them much but when I saw your comment I went back to that one, it's pretty good. It could do with a related but different section in the middle, but apart from that I enjoyed it. Do people ever train at different hierarchical levels? I've done very little ml, but it seems to me that it'd be beneficial to train a net on "plans" and then separately train one to interpret plans.
- craze3 7y ago
  Bare in mind that these exact same MIDI notes can be fed into better sounding instruments (such as lush pads, sharp lead synths, acoustic guitars, etc...), thus improving the overall end product.
  - cthor 7y ago
    Yeah. I think a lot of people are ignoring that the output target, MIDI, is pretty limited. A skilled producer could pretty easily take these exact notes and make a track that sounds great. Sound design makes a huge difference.
    - tartoran 7y ago
      And the constant velocity doesn’t help either. Almost any MIDI score could be made sounding twice as good with a little variation in velocity and slight timing errors. Speaking of which, some friends who had a video production company were picking an audio track from a commercial DVD package and there were song choice in around few dozen different styles to pick from: country, progressive rock, etc. At some point I realized that the same score that was played within different style and textures and different cadence made it sound like different songs altogehter.
carrolldunham 7y ago
the state of this technology after being called 'too dangerous to release'
- est31 7y ago
  1. GPT-2 was built for generating text, not music nor programming languages. So this is pretty impressive. 2. The "too dangerous to release" refers to the text generation mode of GPT-2 and was mainly targeted at spammers etc. 3. Of course there was a hype component in it. They wanted people to start thinking about the question before it's too late, which they did. I doubt that we'll get any kind of attempted world control takeover by AI (unless someone explicitly tells AI to do it, at which point it's a guy trying to seize world control with the help of an AI tool), but even if the chance is very small, if you multiply it with the damage, you should at least start wondering about countermeasures.
- ehsankia 7y ago
  As mentioned in the other comment, this is a very deceitful comment. GPT-2 was made for generating text, and the specific thing it's dangerous for is disinformation bots on twitter/reddit as well as fake generated articles, two things the original model quite excels at. It's like claiming that nuclear bombs aren't dangerous because they're not good at taking us to space.
  - carrolldunham 7y ago
    What I meant by state is status. As in, this type of humble retro styled application is where we find GPT-2 in 2020 instead of dominating the news cycle with what you claim it is so good at. If it excels at text, where are the fake articles and tweets? It's just not that good.
    - ehsankia 7y ago
      By definition, if the tweets and articles are good, they are unnoticeable from real ones. That's exactly what makes them dangerous. If you could detect them, then they wouldn't be dangerous anymore.
p1esk 7y ago
For state of the art in computer generated music check out https://aiva.ai https://aiva.ai
jefftk 7y ago
One thing I was hoping to see from GPT-2 is phrasing. Most of the folk music they're using as input has a structure of internal repetition that lets you know where you are in the tune. Most commonly: AABB structure, 64 beats total, with every 2^n beats clearly grouped. Some of the examples seemed to have this at the 8-beat level, but I couldn't find any with it at the 16, 32, or 64 level. Here's a random snippet of a real tune where you can clearly hear the AABB structure: https://www.jefftk.com/contras/tunes/cast64__starabovethegarter.mp3 https://www.jefftk.com/contras/tunes/cast64__starabovethegar...
- p1esk 7y ago
  You don't really need any AI to do phrase repetition patterns. Typically you simply repeat it as is, or maybe change pitch. This can be done algorithmically after the phrase generating stage. It would be nice if the network learned these patterns (and transitions) on its own, but it's far more important to generate quality phrases. That is the hard part and as you can see from the samples we are not there yet.
  - jefftk 7y ago
    GPT-2 for text is impressive because it does a surprisingly good job with larger scale patterns. If you give it a play it generates things with actor marks, it's sentences are a reasonable length etc. I had been hoping this would be similar to learning the structure of tunes, but it seems that requires something we haven't figured out yet. I do disagree, though, that phrase repetition is as simple as you say. A good 16-count phrase has patterns within it, and a good AABB tune has relationships between the A and B parts.
8bitsrule 7y ago
Probably should have left the title alone. There's no such thing as 'MIDI music'. MIDI is a communications protocol to transfer encoded performance data between electronic sources and sinks. Algorithmic performance data is a thing, but generally not very musical. There've been a couple of exceptions.
- teraflop 7y ago
  The .mid file format is also part of the official MIDI standard. Since that's the format that's ultimately being generated, I don't see what's incorrect or misleading about describing it as "MIDI music". https://en.wikipedia.org/wiki/MIDI#Standard_files https://en.wikipedia.org/wiki/MIDI#Standard_files
- h-cobordism 7y ago
  > There's no such thing as 'MIDI music'. MIDI is a communications protocol to transfer encoded performance data between electronic sources and sinks. Why can't I just as easily say that > There's no such thing as 'sheet music'. Sheets are a communications protocol to transfer encoded performance data between human sources and sinks.
econcon 7y ago
Is there technology available through which we can change voice in real-time on phone calls, I assume for it work latency has to low. There is this project https://github.com/CorentinJ/Real-Time-Voice-Cloning https://github.com/CorentinJ/Real-Time-Voice-Cloning But I found it pretty hard to run, and it doesn't have voice input.
- narrationbox 7y ago
  (Disclaimer: we work on voice synthesis/style transfer and cloning) Depends on how much fidelity you want and how much lag you are willing to accept. Our current voice style transfer state of the art is sufficiently capable though the results may still need anywhere from 6 months to two years of development to be considered "production ready" i.e think poor quality, noise, and artefacts in output audio with existing tech. Pasini. has a pretty good blog post and paper on this: https://towardsdatascience.com/voice-translation-and-audio-style-transfer-with-gans-b63d58f61854 https://towardsdatascience.com/voice-translation-and-audio-s...
slfnflctd 7y ago
Many comments here seem somewhat dismissive, but I heard a number of terrific melodies & counter-melodies and would certainly consider working many of them into part of a song. Also, a lot of the notes or chords which sound 'jarring' to the average listener might work great in a jazz or contemporary classical context, especially with added layering. It's definitely interesting to me. Not sure I see a product in there... but if it could be made more efficient and set up to generate smaller, tighter clips with better instrumentation (after being a trained a bit more on what people like), and had a few key features like time stretching, it could prove useful to creative types. Writing a melody or progression doesn't always come easily, and a little push can do wonders for writer's block.
- snlnspc 7y ago
  This is my take as well. There are a lot of lines that would be right at home in anything moderately jazzy or progressive, and plenty more that I would have a great time using as a starting point.

143 points by gwern 7y ago

mebr 7y ago

The author mentions about 5% of the generated music is any good to take a listen. Even the best sounding ones shared don't sound that good to me.

nr2x 7y ago
Went in with high hopes, but it’s not remotely listenable.
narag 7y ago
Most of it sounds terrible. I guess that it's because there is no way to give some feedback. A chess program knows what means winning. And IIRC "this person doesn't exist" uses human opinion to train. Music has many rules, not only theoretical but unwritten rules about what works. You must incorporate them somehow into the program, either by code or offering something to the program to deduce them.
- deleted 7y ago
  [deleted]
- gwern 7y ago
  I previously tried an approach which uses DRL for feedback but I couldn't quite get it to work: https://www.gwern.net/GPT-2-preference-learning https://www.gwern.net/GPT-2-preference-learning At the moment, it probably would be more practical to train a model to predict ratings and use that to screen generated samples or possible completions and throw out too-low-scoring ones (the 'ranker' approach worked out very well for the Meena chatbot recently: https://arxiv.org/abs/2001.09977 https://arxiv.org/abs/2001.09977 )
  - p1esk 7y ago
    What do you think about training a binary classifier to distinguish between human and generated samples? E.g. choose a state of the art model designed to classify composers or styles, and finetune it for this task.
    - mattkrause 7y ago
      That's a tiny step away from a GAN, which works very well for image generation, so seems promising here too.
      - p1esk 7y ago
        GANs have been tried many times already for music generation, without much success. GPT-2 works very well for text generation so it seemed promising here too. Music falls somewhere in between text (as a sequence of chords or PCM samples) and image (as a piano roll or a spectrogram), so maybe some hybrid of image and text generators is needed.
        TheOtherHobbes 7y ago
        Music does neither, which is why these naive approaches don't work.
    - gwern 7y ago
      I think that could work potentially as a rejection sampling, but you also have the risk that it will simply find some small discriminative detail and be unusefully good at the classification; that's why you do it in a loop as a GAN, but as you mention, GANs work really badly on sequences, still, so... If you wanted to improve my ABC-MIDI GPT-2, the most straightforward ways would be to do data cleaning (I'm sure there's tens of thousands of awful MIDI files which should be removed! data cleaning with RNNs or GPT-2 or GANs always makes a large difference) and increase the model size (the fact that loss bottomed at 0.20, which is still quite bad, suggests that MIDI is hard enough that GPT-2 is struggling). More interesting would be to use Reformer or another long-range Transformer and try to operate directly on a more raw representation, like the the piano roll representation of MIDI. I think GPT-2 makes a lot of syntax errors which cripple outputs when a 'voice' goes silent, and a piano roll representation would be a lot more robust (at the cost of being like 10x larger).
- deleted 7y ago
  [deleted]
- gwern 7y ago
  > And IIRC "this person doesn't exist" uses human opinion to train. It doesn't, incidentally. It's just a standard StyleGAN dumping random images, trained to to model the average image/distribution, and not optimizing for human ratings or anything. Almost all the 'X Does Not Exist' things operate that way, including my own https://www.thiswaifudoesnotexist.net/ https://www.thiswaifudoesnotexist.net/
meatsock 7y ago
the rate among humans who are learning to compose is lower as ideas you don't end up using are silent.
sq_ 7y ago
I thought it was interesting, even if the overall state of most of them wasn't great. Some of them had decent sections, but then there would be some periodically repeated jarring note. Definitely better than I would've expected.
themodelplumber 7y ago
That aspect reminds me of previous experiences with fractal art. If you're evaluating generated sets, there's always going to be a part of an original piece that's off, that's for sure. "Hey, it looks like headlights of cars driving around in the fog! But this other part makes zero sense and is distracting." If you can use a piece as inspiration, browsing around until you find the 5% or whatever % that you like, there's really some benefit to be found. I felt a bit like Simon Cowell, auditioning works and being really picky, last time I did this. But in the end you can discard an item, go with it as-is, adapt it somehow, or use it as reference material. Eventually you build a gallery.
lostcolony 7y ago
Yeah...I'm now mildly curious what could be done using this, plus a genetic algorithm decided either by a human listener, or a suitability function encoding basic ideas in music.
randyrand 7y ago
"LMD: catchy ambient-esque piano piece" is pretty good. I could see it as background music in a game. I like "Pop MIDI, rapid jazz piano?" too.
- kybernetikos 7y ago
  You're right. I listened to a few and didn't like them much but when I saw your comment I went back to that one, it's pretty good. It could do with a related but different section in the middle, but apart from that I enjoyed it. Do people ever train at different hierarchical levels? I've done very little ml, but it seems to me that it'd be beneficial to train a net on "plans" and then separately train one to interpret plans.
craze3 7y ago
Bare in mind that these exact same MIDI notes can be fed into better sounding instruments (such as lush pads, sharp lead synths, acoustic guitars, etc...), thus improving the overall end product.
- cthor 7y ago
  Yeah. I think a lot of people are ignoring that the output target, MIDI, is pretty limited. A skilled producer could pretty easily take these exact notes and make a track that sounds great. Sound design makes a huge difference.
  - tartoran 7y ago
    And the constant velocity doesn’t help either. Almost any MIDI score could be made sounding twice as good with a little variation in velocity and slight timing errors. Speaking of which, some friends who had a video production company were picking an audio track from a commercial DVD package and there were song choice in around few dozen different styles to pick from: country, progressive rock, etc. At some point I realized that the same score that was played within different style and textures and different cadence made it sound like different songs altogehter.

carrolldunham 7y ago

the state of this technology after being called 'too dangerous to release'

est31 7y ago
1. GPT-2 was built for generating text, not music nor programming languages. So this is pretty impressive. 2. The "too dangerous to release" refers to the text generation mode of GPT-2 and was mainly targeted at spammers etc. 3. Of course there was a hype component in it. They wanted people to start thinking about the question before it's too late, which they did. I doubt that we'll get any kind of attempted world control takeover by AI (unless someone explicitly tells AI to do it, at which point it's a guy trying to seize world control with the help of an AI tool), but even if the chance is very small, if you multiply it with the damage, you should at least start wondering about countermeasures.
ehsankia 7y ago
As mentioned in the other comment, this is a very deceitful comment. GPT-2 was made for generating text, and the specific thing it's dangerous for is disinformation bots on twitter/reddit as well as fake generated articles, two things the original model quite excels at. It's like claiming that nuclear bombs aren't dangerous because they're not good at taking us to space.
- carrolldunham 7y ago
  What I meant by state is status. As in, this type of humble retro styled application is where we find GPT-2 in 2020 instead of dominating the news cycle with what you claim it is so good at. If it excels at text, where are the fake articles and tweets? It's just not that good.
  - ehsankia 7y ago
    By definition, if the tweets and articles are good, they are unnoticeable from real ones. That's exactly what makes them dangerous. If you could detect them, then they wouldn't be dangerous anymore.

p1esk 7y ago

For state of the art in computer generated music check out https://aiva.ai https://aiva.ai

jefftk 7y ago

One thing I was hoping to see from GPT-2 is phrasing. Most of the folk music they're using as input has a structure of internal repetition that lets you know where you are in the tune. Most commonly: AABB structure, 64 beats total, with every 2^n beats clearly grouped. Some of the examples seemed to have this at the 8-beat level, but I couldn't find any with it at the 16, 32, or 64 level. Here's a random snippet of a real tune where you can clearly hear the AABB structure: https://www.jefftk.com/contras/tunes/cast64__starabovethegarter.mp3 https://www.jefftk.com/contras/tunes/cast64__starabovethegar...

p1esk 7y ago
You don't really need any AI to do phrase repetition patterns. Typically you simply repeat it as is, or maybe change pitch. This can be done algorithmically after the phrase generating stage. It would be nice if the network learned these patterns (and transitions) on its own, but it's far more important to generate quality phrases. That is the hard part and as you can see from the samples we are not there yet.
- jefftk 7y ago
  GPT-2 for text is impressive because it does a surprisingly good job with larger scale patterns. If you give it a play it generates things with actor marks, it's sentences are a reasonable length etc. I had been hoping this would be similar to learning the structure of tunes, but it seems that requires something we haven't figured out yet. I do disagree, though, that phrase repetition is as simple as you say. A good 16-count phrase has patterns within it, and a good AABB tune has relationships between the A and B parts.

8bitsrule 7y ago

Probably should have left the title alone. There's no such thing as 'MIDI music'. MIDI is a communications protocol to transfer encoded performance data between electronic sources and sinks. Algorithmic performance data is a thing, but generally not very musical. There've been a couple of exceptions.

teraflop 7y ago
The .mid file format is also part of the official MIDI standard. Since that's the format that's ultimately being generated, I don't see what's incorrect or misleading about describing it as "MIDI music". https://en.wikipedia.org/wiki/MIDI#Standard_files https://en.wikipedia.org/wiki/MIDI#Standard_files
h-cobordism 7y ago
> There's no such thing as 'MIDI music'. MIDI is a communications protocol to transfer encoded performance data between electronic sources and sinks. Why can't I just as easily say that > There's no such thing as 'sheet music'. Sheets are a communications protocol to transfer encoded performance data between human sources and sinks.

econcon 7y ago

Is there technology available through which we can change voice in real-time on phone calls, I assume for it work latency has to low. There is this project https://github.com/CorentinJ/Real-Time-Voice-Cloning https://github.com/CorentinJ/Real-Time-Voice-Cloning But I found it pretty hard to run, and it doesn't have voice input.

narrationbox 7y ago
(Disclaimer: we work on voice synthesis/style transfer and cloning) Depends on how much fidelity you want and how much lag you are willing to accept. Our current voice style transfer state of the art is sufficiently capable though the results may still need anywhere from 6 months to two years of development to be considered "production ready" i.e think poor quality, noise, and artefacts in output audio with existing tech. Pasini. has a pretty good blog post and paper on this: https://towardsdatascience.com/voice-translation-and-audio-style-transfer-with-gans-b63d58f61854 https://towardsdatascience.com/voice-translation-and-audio-s...

slfnflctd 7y ago

Many comments here seem somewhat dismissive, but I heard a number of terrific melodies & counter-melodies and would certainly consider working many of them into part of a song. Also, a lot of the notes or chords which sound 'jarring' to the average listener might work great in a jazz or contemporary classical context, especially with added layering. It's definitely interesting to me. Not sure I see a product in there... but if it could be made more efficient and set up to generate smaller, tighter clips with better instrumentation (after being a trained a bit more on what people like), and had a few key features like time stretching, it could prove useful to creative types. Writing a melody or progression doesn't always come easily, and a little push can do wonders for writer's block.

snlnspc 7y ago
This is my take as well. There are a lot of lines that would be right at home in anything moderately jazzy or progressive, and plenty more that I would have a great time using as a starting point.