7 ms·

Training a Recurrent Neural Network to Compose Music

134 points by dacm 10y ago

minimaxir 10y ago
Another (hilarious) tool which demonstrates the difference in quality of lightly-trained RNNs and strongly-trained RNNs is RoboRosewater, which generates Magic: the Gathering cards using networks of varying quality/sanity, indicated by the card art: https://twitter.com/RoboRosewater https://twitter.com/RoboRosewater
- jjcm 10y ago
  Along those lines, the markov chain dota 2 patch notes: https://www.reddit.com/r/DotA2/comments/4fw6gm/how_687_would_look_like_if_icefrog_was_a_markov/ https://www.reddit.com/r/DotA2/comments/4fw6gm/how_687_would... The best part is Valve actually implemented one of them: http://i.imgur.com/Ydim1ui.png http://i.imgur.com/Ydim1ui.png
yomly 10y ago
The music came out quite fun sounding. I could almost imagine hearing at least passages within the music in videogames. Perhaps some old-school zelda/jrpg game which would suit the folky quality of the music. The bass line was generally quite simplistic, I wonder what happens if you codified gradus ad parnassum and taught the RNN counterpoint[0] [0] https://en.m.wikipedia.org/wiki/Johann_Joseph_Fux https://en.m.wikipedia.org/wiki/Johann_Joseph_Fux
- TheOtherHobbes 10y ago
  Fux wouldn't apply here because the bass/chord accompaniment shapes are idiomatic, and Fuxian counterpoint wouldn't be. In fact this kind of toy ML can never produce competent music. The best it can do is produce short workable snatches that sound like cut and paste snippets of the training source - before losing the plot in the next bar or two. Training an RNN on a huge set of tunes and expecting it to produce examples of equivalent musicality is a fundamentally unworkable idea. I'd suggest that anyone who doesn't see why this must be true doesn't understand ML well enough to know when it can and can't be used effectively. It's worth asking in what other domains are trivial RNNs being misapplied to produce trivially poor models. It's one thing to make bad music. It's another to - say - run a trading strategy, or make marketing decisions based on oversimplified ML models that produce misleading results because they're not sophisticated enough to recognise all the critical structures in the data set.
empath75 10y ago
Is there a better music database to work from for music generation? I'm surprised there isn't a massive db of 19th century sheet music or player piano rolls somewhere.
- jcranmer 10y ago
  Musescore has a user database that has lots of music (no song count that I could find quickly, but I want to say ~10s-100s of thousands) that ought to be convertible to appropriate formats. IMSLP (<http://imslp.org/wiki/Main_Page> http://imslp.org/wiki/Main_Page>) has about 110,000 works uploaded. However, these are mostly PDFs (some of them are scans), which would be very hard to extract useful data from. Such data may include multiple voices, which makes it harder for a neural network to learn a pleasant-sounding song.
  - Animats 10y ago
    Try taking in MIDI files. There are lots of those, and they have all the information of a score.
- lawless123 10y ago
  Does it have to be sheet music? are there legal or technical reasons i couldn't just use loads of my mp3's?
Question1101 10y ago
I am a neural network noob and only know the basic feedforward network. So the training set is just text files containing songs? How does it test if the output is correct or not? If I understand correctly the goal here was just to produce outputs in the correct format. If one wanted to train for quality as well would one need to grade every output the network produces by hand?
- visarga 10y ago
  In the training phase the network learns to predict the next note from the past few beats. As training data it uses a bunch of text formatted midi files. In generative mode, they just feed back the generated notes into the network at each step. The same approach can be used to generate DeepDrumpf[1] or any kind of time series. [1] https://twitter.com/deepdrumpf https://twitter.com/deepdrumpf Funny, this is DeepDrumpf bad mouthing Andrej Karpathy, the inventor of this algorithm (char-rnn): https://twitter.com/karpathy/status/705558159964803072 https://twitter.com/karpathy/status/705558159964803072
  - wodenokoto 10y ago
    I believe Karpathy Char-RNN is from 2015 (that is at least the year on the 100-line gist and the "unreasonable effectiveness of ..." blog post[0]) but char-level RNN language models dates back to at least 2011 with [1] So I don't think we should call him the inventor, though he definitely popularised it with his great writing and examples. [0] http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/ [1] Sutskever, Ilya, James Martens, and Geoffrey E. Hinton. "Generating text with recurrent neural networks." Proceedings of the 28th International Conference on Machine Learning (ICML-11). 2011.
  - slagfart 10y ago
    I'd love to see DeepDril, derived from https://twitter.com/dril https://twitter.com/dril Here's all his tweets: http://greptweet.com/u/dril/dril.txt http://greptweet.com/u/dril/dril.txt
xylophoner 10y ago
This isn't composition. Mimicry, perhaps.
- zump 10y ago
  Shots fired..
- deckar01 10y ago
  The patterns that RNNs tease out of training data seem to transcend mimicry. An artist may compose a song that sounds just like another song, but it is not simply mimicry, it is influence. It is a learned set of patterns that are composed somewhat randomly. The human mind has this illusion that what it hears is subjectively good, but that intuition is built out of experiences just like the RNN.
  - vixen99 10y ago
    How can you have an illusion that something is subjectively good?
lopatin 10y ago
Noob question: It intuitively seems to me that by feeding raw text of a structured format (such as the music notation in the article) we're making the algorithm unnecessarily learn the syntax in addition to interesting stuff, which is the high level musical patterns. What kind of results would you expect from running the same experiment, but with an input encoding more specialized to the problem domain? Would the performance benefits be significant?
- adamnemecek 10y ago
  Part of the appeal of ml is that you don't have to care about the underlying processes.
  - lopatin 10y ago
    We want the model to be better right? I just want to express input data using as little bytes as possible. Do you mean that if we are able to somehow express in 1 byte what used to take 100, that you wouldn't expect it to noticeably improve the model?
    - azernik 10y ago
      Not appreciably. Neural Networks' job (in some configurations) is to approximate their input in as few bytes as possible. There's a vague equivalence between machine learning and lossy compression.
      - Scea91 10y ago
        There's even vague equivalence between statistics and lossy compression.
        azernik 10y ago
        And most modern machine learning is referred to in academia as "statistical learning". Equivalences all the way down!
lubujackson 10y ago
If you think this is impressive, consider that Mozart himself wrote an algorithm to randomly generator music... and it still sounds like Mozart: http://www.rationalargumentator.com/index/blog/2015/06/variations-minuet-mozart/ http://www.rationalargumentator.com/index/blog/2015/06/varia...
- anoplus 10y ago
  I always felt Mozart had some kind of a formula that made him so productive in his short life. Now I read he had. For MOZARTish music at least.
franze 10y ago
So has somebody fed all (i.e.: JS) github code into a neural network yet?
- augustt 10y ago
  Not quite at that scale, but Andrej Karpathy fed the linux source code into an RNN: http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/
Animats 10y ago
Isn't this the third project like this on HN? These things generate tunes which sound OK for a few seconds, but after tens of seconds, you realize there's no higher level structure at all. It's just random.
- dimmuborgir 10y ago
  Google's Magenta is exploring generation of long-term structure in music. [1] Its demo song is fun [2] [1] - https://magenta.tensorflow.org/2016/07/15/lookback-rnn-attention-rnn/ https://magenta.tensorflow.org/2016/07/15/lookback-rnn-atten... [2] - https://www.youtube.com/watch?v=qFBQDfPyjoE https://www.youtube.com/watch?v=qFBQDfPyjoE
jaza 10y ago
The authors of these music-generators should submit some of the compositions to online music libraries, to song competitions, etc, and see if they get accepted! "A la" what happened, back in the day, with peer-reviewed journals and: http://www.elsewhere.org/journal/pomo/ http://www.elsewhere.org/journal/pomo/
jcraane 10y ago
A whole time back I created a program which uses genetic algorithms to generate melodies. I used the generated melodies as inspiration for music composition. The idea was (never implemented though) to add a fitness function based on training a neural network which could be trained by looking at other melodies or user input. More information can be found here: http://jcraane.blogspot.nl/2009/06/melody-composition-using-genetic.html http://jcraane.blogspot.nl/2009/06/melody-composition-using-... The source code is available here: https://github.com/jcraane/melodycomposition_genetic https://github.com/jcraane/melodycomposition_genetic Some sample melodies are in de docs/samples folder. I may cost some time to get it to work again but should not be that hard.

134 points by dacm 10y ago

Another (hilarious) tool which demonstrates the difference in quality of lightly-trained RNNs and strongly-trained RNNs is RoboRosewater, which generates Magic: the Gathering cards using networks of varying quality/sanity, indicated by the card art: https://twitter.com/RoboRosewater https://twitter.com/RoboRosewater

jjcm 10y ago
Along those lines, the markov chain dota 2 patch notes: https://www.reddit.com/r/DotA2/comments/4fw6gm/how_687_would_look_like_if_icefrog_was_a_markov/ https://www.reddit.com/r/DotA2/comments/4fw6gm/how_687_would... The best part is Valve actually implemented one of them: http://i.imgur.com/Ydim1ui.png http://i.imgur.com/Ydim1ui.png

yomly 10y ago

The music came out quite fun sounding. I could almost imagine hearing at least passages within the music in videogames. Perhaps some old-school zelda/jrpg game which would suit the folky quality of the music. The bass line was generally quite simplistic, I wonder what happens if you codified gradus ad parnassum and taught the RNN counterpoint[0] [0] https://en.m.wikipedia.org/wiki/Johann_Joseph_Fux https://en.m.wikipedia.org/wiki/Johann_Joseph_Fux

TheOtherHobbes 10y ago
Fux wouldn't apply here because the bass/chord accompaniment shapes are idiomatic, and Fuxian counterpoint wouldn't be. In fact this kind of toy ML can never produce competent music. The best it can do is produce short workable snatches that sound like cut and paste snippets of the training source - before losing the plot in the next bar or two. Training an RNN on a huge set of tunes and expecting it to produce examples of equivalent musicality is a fundamentally unworkable idea. I'd suggest that anyone who doesn't see why this must be true doesn't understand ML well enough to know when it can and can't be used effectively. It's worth asking in what other domains are trivial RNNs being misapplied to produce trivially poor models. It's one thing to make bad music. It's another to - say - run a trading strategy, or make marketing decisions based on oversimplified ML models that produce misleading results because they're not sophisticated enough to recognise all the critical structures in the data set.

empath75 10y ago

Is there a better music database to work from for music generation? I'm surprised there isn't a massive db of 19th century sheet music or player piano rolls somewhere.

jcranmer 10y ago
Musescore has a user database that has lots of music (no song count that I could find quickly, but I want to say ~10s-100s of thousands) that ought to be convertible to appropriate formats. IMSLP (<http://imslp.org/wiki/Main_Page> http://imslp.org/wiki/Main_Page>) has about 110,000 works uploaded. However, these are mostly PDFs (some of them are scans), which would be very hard to extract useful data from. Such data may include multiple voices, which makes it harder for a neural network to learn a pleasant-sounding song.
- Animats 10y ago
  Try taking in MIDI files. There are lots of those, and they have all the information of a score.
lawless123 10y ago
Does it have to be sheet music? are there legal or technical reasons i couldn't just use loads of my mp3's?

Question1101 10y ago

I am a neural network noob and only know the basic feedforward network. So the training set is just text files containing songs? How does it test if the output is correct or not? If I understand correctly the goal here was just to produce outputs in the correct format. If one wanted to train for quality as well would one need to grade every output the network produces by hand?

visarga 10y ago
In the training phase the network learns to predict the next note from the past few beats. As training data it uses a bunch of text formatted midi files. In generative mode, they just feed back the generated notes into the network at each step. The same approach can be used to generate DeepDrumpf[1] or any kind of time series. [1] https://twitter.com/deepdrumpf https://twitter.com/deepdrumpf Funny, this is DeepDrumpf bad mouthing Andrej Karpathy, the inventor of this algorithm (char-rnn): https://twitter.com/karpathy/status/705558159964803072 https://twitter.com/karpathy/status/705558159964803072
- wodenokoto 10y ago
  I believe Karpathy Char-RNN is from 2015 (that is at least the year on the 100-line gist and the "unreasonable effectiveness of ..." blog post[0]) but char-level RNN language models dates back to at least 2011 with [1] So I don't think we should call him the inventor, though he definitely popularised it with his great writing and examples. [0] http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/ [1] Sutskever, Ilya, James Martens, and Geoffrey E. Hinton. "Generating text with recurrent neural networks." Proceedings of the 28th International Conference on Machine Learning (ICML-11). 2011.
- slagfart 10y ago
  I'd love to see DeepDril, derived from https://twitter.com/dril https://twitter.com/dril Here's all his tweets: http://greptweet.com/u/dril/dril.txt http://greptweet.com/u/dril/dril.txt

xylophoner 10y ago

This isn't composition. Mimicry, perhaps.

zump 10y ago
Shots fired..
deckar01 10y ago
The patterns that RNNs tease out of training data seem to transcend mimicry. An artist may compose a song that sounds just like another song, but it is not simply mimicry, it is influence. It is a learned set of patterns that are composed somewhat randomly. The human mind has this illusion that what it hears is subjectively good, but that intuition is built out of experiences just like the RNN.
- vixen99 10y ago
  How can you have an illusion that something is subjectively good?

lopatin 10y ago

Noob question: It intuitively seems to me that by feeding raw text of a structured format (such as the music notation in the article) we're making the algorithm unnecessarily learn the syntax in addition to interesting stuff, which is the high level musical patterns. What kind of results would you expect from running the same experiment, but with an input encoding more specialized to the problem domain? Would the performance benefits be significant?

adamnemecek 10y ago
Part of the appeal of ml is that you don't have to care about the underlying processes.
- lopatin 10y ago
  We want the model to be better right? I just want to express input data using as little bytes as possible. Do you mean that if we are able to somehow express in 1 byte what used to take 100, that you wouldn't expect it to noticeably improve the model?
  - azernik 10y ago
    Not appreciably. Neural Networks' job (in some configurations) is to approximate their input in as few bytes as possible. There's a vague equivalence between machine learning and lossy compression.
    - Scea91 10y ago
      There's even vague equivalence between statistics and lossy compression.
      - azernik 10y ago
        And most modern machine learning is referred to in academia as "statistical learning". Equivalences all the way down!

lubujackson 10y ago

If you think this is impressive, consider that Mozart himself wrote an algorithm to randomly generator music... and it still sounds like Mozart: http://www.rationalargumentator.com/index/blog/2015/06/variations-minuet-mozart/ http://www.rationalargumentator.com/index/blog/2015/06/varia...

anoplus 10y ago
I always felt Mozart had some kind of a formula that made him so productive in his short life. Now I read he had. For MOZARTish music at least.

franze 10y ago

So has somebody fed all (i.e.: JS) github code into a neural network yet?

augustt 10y ago
Not quite at that scale, but Andrej Karpathy fed the linux source code into an RNN: http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/

Animats 10y ago

Isn't this the third project like this on HN? These things generate tunes which sound OK for a few seconds, but after tens of seconds, you realize there's no higher level structure at all. It's just random.

dimmuborgir 10y ago
Google's Magenta is exploring generation of long-term structure in music. [1] Its demo song is fun [2] [1] - https://magenta.tensorflow.org/2016/07/15/lookback-rnn-attention-rnn/ https://magenta.tensorflow.org/2016/07/15/lookback-rnn-atten... [2] - https://www.youtube.com/watch?v=qFBQDfPyjoE https://www.youtube.com/watch?v=qFBQDfPyjoE

jaza 10y ago

The authors of these music-generators should submit some of the compositions to online music libraries, to song competitions, etc, and see if they get accepted! "A la" what happened, back in the day, with peer-reviewed journals and: http://www.elsewhere.org/journal/pomo/ http://www.elsewhere.org/journal/pomo/

jcraane 10y ago

A whole time back I created a program which uses genetic algorithms to generate melodies. I used the generated melodies as inspiration for music composition. The idea was (never implemented though) to add a fitness function based on training a neural network which could be trained by looking at other melodies or user input. More information can be found here: http://jcraane.blogspot.nl/2009/06/melody-composition-using-genetic.html http://jcraane.blogspot.nl/2009/06/melody-composition-using-... The source code is available here: https://github.com/jcraane/melodycomposition_genetic https://github.com/jcraane/melodycomposition_genetic Some sample melodies are in de docs/samples folder. I may cost some time to get it to work again but should not be that hard.