6 ms·

LSTM Neural Network that tries to write piano melodies similar to Bach's (2016)

75 points by maxwell 8y ago

abhishekjha 8y ago
A file with space in its name? We had an entire article about not to do this here.
p1esk 8y ago
30 LSTM layers of 30 neurons each? Two notes as input? Those are some unusual choices... For a dataset this small, a single layer LSTM with 300 neurons, accepting four bars of input at a time, would most likely do much better.
sabertoothed 8y ago
From 2016. Add that year to the title.
partycoder 8y ago
Very different to Bach.
stevehiehn 8y ago
Very cool, you might be interested in an open source chord-melody dataset I’m hand curating for generating pop melodies: https://github.com/shiehn/chord-melody-dataset https://github.com/shiehn/chord-melody-dataset. I’m feeding into LSTM’s and and its working quite well in my system :) http://treblemaker.ai/ http://treblemaker.ai/
kastnerkyle 8y ago
Glad to see people posting links to music modeling projects - it's a huge space, and a ton of fun to actively work on or just mess with if you like music and code. Even simple markov models can get someplace with a bit of tweaking [0][1]. If you like this, you might like some more recent work from Anna huang et. al. at Magenta, using a modified Transformer network [2][3]. Some other relevant reading includes DeepBach [4][5], and outside the domain of Bach alone, the venerable folkRNN [6]. There's a huge chain of work I posted a while back here [7], and Kyle McDonald wrote a really great review piece here [8]. Modeling music in latent spaces has also seen a number of recent publications [9][10][11], and having smooth latent spaces for generation seems (to me) to be a very natural interface for manipulating things, which has a natural analogue in "real" composition, cool video at [12]. There have also been some really fun online demos for interacting with these types of systems [13] There's a lot of interesting work going on in music modeling, hopefully leading to new and interesting musical tools and toys. Interactivity (often via clever conditioning) can really change the way you approach these types of problems [14]. [0] https://github.com/kastnerkyle/magenta/tree/my_cleanup_branch/exp/markov_lm https://github.com/kastnerkyle/magenta/tree/my_cleanup_branc... [1] https://www.youtube.com/watch?v=Ay8mHW0-NW8&list=PLRMa_gJ8vx8mC9VvtXE7eGamGdf3MxzV4 https://www.youtube.com/watch?v=Ay8mHW0-NW8&list=PLRMa_gJ8vx... [2] https://twitter.com/kastnerkyle/status/1018617315196309504 https://twitter.com/kastnerkyle/status/1018617315196309504 [3] https://arxiv.org/abs/1809.04281 https://arxiv.org/abs/1809.04281 [4] https://www.flow-machines.com/archives/deepbach-polyphonic-music-generation-bach-chorales/ https://www.flow-machines.com/archives/deepbach-polyphonic-m... [5] https://github.com/Ghadjeres/DeepBach https://github.com/Ghadjeres/DeepBach [6] https://github.com/IraKorshunova/folk-rnn https://github.com/IraKorshunova/folk-rnn [7] https://news.ycombinator.com/item?id=14822281 https://news.ycombinator.com/item?id=14822281 [8] https://medium.com/artists-and-machine-intelligence/neural-nets-for-generating-music-f46dffac21c0 https://medium.com/artists-and-machine-intelligence/neural-n... [9] https://storage.googleapis.com/magentadata/papers/multitrack/index.html https://storage.googleapis.com/magentadata/papers/multitrack... [10] https://magenta.tensorflow.org/music-vae https://magenta.tensorflow.org/music-vae [11] https://www.youtube.com/watch?v=BbyvbO2F7ug https://www.youtube.com/watch?v=BbyvbO2F7ug [12] https://www.youtube.com/watch?v=A8d-PUuUJnA https://www.youtube.com/watch?v=A8d-PUuUJnA [13] https://codepen.io/teropa/ https://codepen.io/teropa/ [14] https://twitter.com/chrisdonahuey/status/1051868153700016128 https://twitter.com/chrisdonahuey/status/1051868153700016128
Jagat 8y ago
Btw I just noticed that he's the same guy whose work on GAN was used by an art collective, and the resulting image sold for $14k. https://twitter.com/DrBeef_/status/1055285640420483073 https://twitter.com/DrBeef_/status/1055285640420483073
- dplavery92 8y ago
  $432k, actually! https://www.nytimes.com/2018/10/25/arts/design/ai-art-sold-christies.html https://www.nytimes.com/2018/10/25/arts/design/ai-art-sold-c...
halfvoxel 8y ago
Cool! Music generated by neural networks is such an interesting field. Me and some friends also had a go at it. We used two different datasets, one from Bach and one from Final Fantasy. In contrast to many other approaches, we allowed chords as well as playing multiple notes at the same time (not necessarily the same as playing a chord). You can find some results here: https://github.com/HalfVoxel/deep-piano https://github.com/HalfVoxel/deep-piano (they are definitely cherry-picked, but it worked out quite well overall imo).
muxator 8y ago
Nice idea. I suspect Bach made use of a lot more neurons to write this: https://m.youtube.com/watch?v=aEkXet4WX_c&t=7s https://m.youtube.com/watch?v=aEkXet4WX_c&t=7s
saaaaaam 8y ago
I’m not sure why this has been compared to Bach. It’s not very similar at all, and quite jarringly disorganised. I can’t comment on the method used to make it (though would love to understand more, as this is an area that professionally interests me) but you may want to go back to the drawing board :)
sterlind 8y ago
A Markov chain would have done better. Something is seriously wrong here - not even the 2-grams are accurate. If your neural network can't outperform a simple control like a Markov chain, it's hardly worth publishing.

75 points by maxwell 8y ago

A file with space in its name? We had an entire article about not to do this here.

30 LSTM layers of 30 neurons each? Two notes as input? Those are some unusual choices... For a dataset this small, a single layer LSTM with 300 neurons, accepting four bars of input at a time, would most likely do much better.

sabertoothed 8y ago

From 2016. Add that year to the title.

partycoder 8y ago

Very different to Bach.

stevehiehn 8y ago

Very cool, you might be interested in an open source chord-melody dataset I’m hand curating for generating pop melodies: https://github.com/shiehn/chord-melody-dataset https://github.com/shiehn/chord-melody-dataset. I’m feeding into LSTM’s and and its working quite well in my system :) http://treblemaker.ai/ http://treblemaker.ai/

kastnerkyle 8y ago

Glad to see people posting links to music modeling projects - it's a huge space, and a ton of fun to actively work on or just mess with if you like music and code. Even simple markov models can get someplace with a bit of tweaking [0][1]. If you like this, you might like some more recent work from Anna huang et. al. at Magenta, using a modified Transformer network [2][3]. Some other relevant reading includes DeepBach [4][5], and outside the domain of Bach alone, the venerable folkRNN [6]. There's a huge chain of work I posted a while back here [7], and Kyle McDonald wrote a really great review piece here [8]. Modeling music in latent spaces has also seen a number of recent publications [9][10][11], and having smooth latent spaces for generation seems (to me) to be a very natural interface for manipulating things, which has a natural analogue in "real" composition, cool video at [12]. There have also been some really fun online demos for interacting with these types of systems [13] There's a lot of interesting work going on in music modeling, hopefully leading to new and interesting musical tools and toys. Interactivity (often via clever conditioning) can really change the way you approach these types of problems [14]. [0] https://github.com/kastnerkyle/magenta/tree/my_cleanup_branch/exp/markov_lm https://github.com/kastnerkyle/magenta/tree/my_cleanup_branc... [1] https://www.youtube.com/watch?v=Ay8mHW0-NW8&list=PLRMa_gJ8vx8mC9VvtXE7eGamGdf3MxzV4 https://www.youtube.com/watch?v=Ay8mHW0-NW8&list=PLRMa_gJ8vx... [2] https://twitter.com/kastnerkyle/status/1018617315196309504 https://twitter.com/kastnerkyle/status/1018617315196309504 [3] https://arxiv.org/abs/1809.04281 https://arxiv.org/abs/1809.04281 [4] https://www.flow-machines.com/archives/deepbach-polyphonic-music-generation-bach-chorales/ https://www.flow-machines.com/archives/deepbach-polyphonic-m... [5] https://github.com/Ghadjeres/DeepBach https://github.com/Ghadjeres/DeepBach [6] https://github.com/IraKorshunova/folk-rnn https://github.com/IraKorshunova/folk-rnn [7] https://news.ycombinator.com/item?id=14822281 https://news.ycombinator.com/item?id=14822281 [8] https://medium.com/artists-and-machine-intelligence/neural-nets-for-generating-music-f46dffac21c0 https://medium.com/artists-and-machine-intelligence/neural-n... [9] https://storage.googleapis.com/magentadata/papers/multitrack/index.html https://storage.googleapis.com/magentadata/papers/multitrack... [10] https://magenta.tensorflow.org/music-vae https://magenta.tensorflow.org/music-vae [11] https://www.youtube.com/watch?v=BbyvbO2F7ug https://www.youtube.com/watch?v=BbyvbO2F7ug [12] https://www.youtube.com/watch?v=A8d-PUuUJnA https://www.youtube.com/watch?v=A8d-PUuUJnA [13] https://codepen.io/teropa/ https://codepen.io/teropa/ [14] https://twitter.com/chrisdonahuey/status/1051868153700016128 https://twitter.com/chrisdonahuey/status/1051868153700016128

Jagat 8y ago

Btw I just noticed that he's the same guy whose work on GAN was used by an art collective, and the resulting image sold for $14k. https://twitter.com/DrBeef_/status/1055285640420483073 https://twitter.com/DrBeef_/status/1055285640420483073

dplavery92 8y ago
$432k, actually! https://www.nytimes.com/2018/10/25/arts/design/ai-art-sold-christies.html https://www.nytimes.com/2018/10/25/arts/design/ai-art-sold-c...

halfvoxel 8y ago

Cool! Music generated by neural networks is such an interesting field. Me and some friends also had a go at it. We used two different datasets, one from Bach and one from Final Fantasy. In contrast to many other approaches, we allowed chords as well as playing multiple notes at the same time (not necessarily the same as playing a chord). You can find some results here: https://github.com/HalfVoxel/deep-piano https://github.com/HalfVoxel/deep-piano (they are definitely cherry-picked, but it worked out quite well overall imo).

muxator 8y ago

Nice idea. I suspect Bach made use of a lot more neurons to write this: https://m.youtube.com/watch?v=aEkXet4WX_c&t=7s https://m.youtube.com/watch?v=aEkXet4WX_c&t=7s

saaaaaam 8y ago

I’m not sure why this has been compared to Bach. It’s not very similar at all, and quite jarringly disorganised. I can’t comment on the method used to make it (though would love to understand more, as this is an area that professionally interests me) but you may want to go back to the drawing board :)

sterlind 8y ago

A Markov chain would have done better. Something is seriously wrong here - not even the 2-grams are accurate. If your neural network can't outperform a simple control like a Markov chain, it's hardly worth publishing.