4 ms·

Re: the latest Performance RNN, discussing their "best sounding" output. >Note that this isn’t a performance of an existing piece; the model is also choosing t

Re: the latest Performance RNN, discussing their "best sounding" output. >Note that this isn’t a performance of an existing piece; the model is also choosing the notes to play, “composing” a performance directly. The performances generated by the model lack the overall coherence that one might expect from a piano composition; in musical jargon, it might sound like the model is “noodling”— playing without a long-term structure. However, to our ears, the local characteristics of the performance (i.e. the phrasing within a one or two second time window) are quite expressive. It's understandable to be encouraged by progress, but to my ears, it's not really expressive other than the first thought that I had when listening to the piece: "That piano player is having a stroke, somebody should do something to help."

kastnerkyle 9y ago
My take (as someone working in the area) on how/why we are encouraged by this progress: The model is unconditional i.e. any concept of form at all was learned directly from data, with no real structural hints to what it should learn. Generally you get much stronger "global structure" the more prior information you provide either in the model itself or in the input/targets (such as chord constraints, etc.). This is one reason that harmonization tasks often sound notably better than pure generation - the "backbone" in harmonization was human provided, whereas in pure generation the model needs to be self-consistent. The analogous task in language would be character level language modeling, whereas something like harmonization or conditional generation from chords would be more akin to machine translation. Character level language models may be a bit wonky looking when generating, but any structure at all was purely model discovered from a simple rule (generally, maximum likelihood), whereas in conditional models a lot of extra help is given by the conditioning variable. This model is a pretty big jump in quality from previous efforts in the same vein of unconditional generation, and should combine well with techniques for conditioning in sequence models from all over the research community (translation, speech recognition, question answering, summarization, speech synthesis). On top of that, the model is IMO quite elegant, and much simpler to understand than many other attempts at polyphonic generation including my own. It is worth comparing this output to some previous efforts (from Magenta and others), some with more complicated internal structure such as [0][1][2][3][4][5][6], and also some models with stronger conditioning or different input representations such as [7][8][9], the extreme case being [10] where a human (Benoît Carré) used a modeling/ML toolkit along with standard arrangement and production approaches to produce a wacky, awesome pop song. Magenta is putting out lots of neat stuff around interactive usage of these models, with web demos and plugins to standard music tools [11]. Indeed, Doug's quotes in this article [12] sum it up for me, excerpt: “I don’t think that machines themselves just making art for art’s sake is as interesting as you might think,” he explained. “The question to ask is can machines help us make a new kind of art?” Link to the model in question: Performance RNN, Magenta https://magenta.tensorflow.org/performance-rnn https://magenta.tensorflow.org/performance-rnn [0] Work from Doug in his postdoc doing LSTM Blues, http://people.idsia.ch/~juergen/blues/ http://people.idsia.ch/~juergen/blues/ [1] RNN-RBM from Boulanger-Lewandowski et. al, see TF impl http://danshiebler.com/2016-08-17-musical-tensorflow-part-two-the-rnn-rbm/ http://danshiebler.com/2016-08-17-musical-tensorflow-part-tw... [2] Early Magenta models, https://magenta.tensorflow.org/2016/07/15/lookback-rnn-attention-rnn https://magenta.tensorflow.org/2016/07/15/lookback-rnn-atten... , ex https://www.youtube.com/watch?v=nHxr9u9_4_s https://www.youtube.com/watch?v=nHxr9u9_4_s [3] Daniel Johnson's Biaxial RNN http://www.hexahedria.com/2015/08/03/composing-music-with-recurrent-neural-networks/ http://www.hexahedria.com/2015/08/03/composing-music-with-re... [4] Some experiments only published in my thesis to date, https://www.youtube.com/watch?v=tavHJoum--g&list=PLRMa_gJ8vx8mKp_5pcmh7YFnbTiD8BsQO&index=4 https://www.youtube.com/watch?v=tavHJoum--g&list=PLRMa_gJ8vx... , thesis in question https://github.com/kastnerkyle/udem_masters_thesis https://github.com/kastnerkyle/udem_masters_thesis [5] Polyphonic Magenta model https://www.youtube.com/watch?v=s0BVFVqEY4A https://www.youtube.com/watch?v=s0BVFVqEY4A [6] RL tuning to improve generation https://magenta.tensorflow.org/2016/11/09/tuning-recurrent-networks-with-reinforcement-learning https://magenta.tensorflow.org/2016/11/09/tuning-recurrent-n... [7] Folk-RNN is similar but uses ABC representation (https://highnoongmt.wordpress.com/2015/05/22/lisls-stis-recurrent-neural-networks-for-folk-music-generation/ https://highnoongmt.wordpress.com/2015/05/22/lisls-stis-recu...), https://soundcloud.com/seaandsailor/sets/char-rnn-composes-irish-folk-music https://soundcloud.com/seaandsailor/sets/char-rnn-composes-i..., also see related post https://maraoz.com/2016/02/02/abc-rnn/ https://maraoz.com/2016/02/02/abc-rnn/ [8] DeepBach , http://www.flow-machines.com/deepbach-polyphonic-music-generation-bach-chorales/ http://www.flow-machines.com/deepbach-polyphonic-music-gener... [9] Lawson He's LSTM approach, http://www.lawsonhe.com/music.html http://www.lawsonhe.com/music.html [10] Daddy's Car https://www.youtube.com/watch?v=LSHZ_b05W7o https://www.youtube.com/watch?v=LSHZ_b05W7o [11] Interactive Magenta Jam https://magenta.tensorflow.org/2016/12/16/nips-demo https://magenta.tensorflow.org/2016/12/16/nips-demo [12] https://www.technologyreview.com/s/604010/google-brain-wants-creative-ai-to-help-humans-make-a-new-kind-of-art/ https://www.technologyreview.com/s/604010/google-brain-wants...
- dougeck 9y ago
  Wow. Better bib than we ourselves have! :)
deleted 9y ago
[deleted]
dougeck 9y ago
I've used the same analogy. Its unconditional and produces no coherent structure, but at least it captures some pianist-like timing and dynamics. Magenta is a github project ... please do something to help! :-)
- 6stringmerc 9y ago
  Eh, asking a musician to write code is like asking a programmer to learn a scale on an accordion. Either you're going to see the failures in your own art, and be open to criticism, or ask for somebody to do it for you. I'll put it thusly: As a broke-ass musician barely making ends meet, I'll be more than happy to contribute if Alphabet pays off what's left of the mortgage on my house. Until then, I've only got the perspective that a project like this is trying to take food off my plate, money out of my pocket, and replace me. If you want my help, Son, you got to come better at me than with Altruism. Otherwise, the best thing I can do to help is to say "This project is a heap of shit and your time is better spent elsewhere" without tinging it with too much condescension. Your child is brain damaged. Take it to Oregon.

kastnerkyle 9y ago

My take (as someone working in the area) on how/why we are encouraged by this progress: The model is unconditional i.e. any concept of form at all was learned directly from data, with no real structural hints to what it should learn. Generally you get much stronger "global structure" the more prior information you provide either in the model itself or in the input/targets (such as chord constraints, etc.). This is one reason that harmonization tasks often sound notably better than pure generation - the "backbone" in harmonization was human provided, whereas in pure generation the model needs to be self-consistent. The analogous task in language would be character level language modeling, whereas something like harmonization or conditional generation from chords would be more akin to machine translation. Character level language models may be a bit wonky looking when generating, but any structure at all was purely model discovered from a simple rule (generally, maximum likelihood), whereas in conditional models a lot of extra help is given by the conditioning variable. This model is a pretty big jump in quality from previous efforts in the same vein of unconditional generation, and should combine well with techniques for conditioning in sequence models from all over the research community (translation, speech recognition, question answering, summarization, speech synthesis). On top of that, the model is IMO quite elegant, and much simpler to understand than many other attempts at polyphonic generation including my own. It is worth comparing this output to some previous efforts (from Magenta and others), some with more complicated internal structure such as [0][1][2][3][4][5][6], and also some models with stronger conditioning or different input representations such as [7][8][9], the extreme case being [10] where a human (Benoît Carré) used a modeling/ML toolkit along with standard arrangement and production approaches to produce a wacky, awesome pop song. Magenta is putting out lots of neat stuff around interactive usage of these models, with web demos and plugins to standard music tools [11]. Indeed, Doug's quotes in this article [12] sum it up for me, excerpt: “I don’t think that machines themselves just making art for art’s sake is as interesting as you might think,” he explained. “The question to ask is can machines help us make a new kind of art?” Link to the model in question: Performance RNN, Magenta https://magenta.tensorflow.org/performance-rnn https://magenta.tensorflow.org/performance-rnn [0] Work from Doug in his postdoc doing LSTM Blues, http://people.idsia.ch/~juergen/blues/ http://people.idsia.ch/~juergen/blues/ [1] RNN-RBM from Boulanger-Lewandowski et. al, see TF impl http://danshiebler.com/2016-08-17-musical-tensorflow-part-two-the-rnn-rbm/ http://danshiebler.com/2016-08-17-musical-tensorflow-part-tw... [2] Early Magenta models, https://magenta.tensorflow.org/2016/07/15/lookback-rnn-attention-rnn https://magenta.tensorflow.org/2016/07/15/lookback-rnn-atten... , ex https://www.youtube.com/watch?v=nHxr9u9_4_s https://www.youtube.com/watch?v=nHxr9u9_4_s [3] Daniel Johnson's Biaxial RNN http://www.hexahedria.com/2015/08/03/composing-music-with-recurrent-neural-networks/ http://www.hexahedria.com/2015/08/03/composing-music-with-re... [4] Some experiments only published in my thesis to date, https://www.youtube.com/watch?v=tavHJoum--g&list=PLRMa_gJ8vx8mKp_5pcmh7YFnbTiD8BsQO&index=4 https://www.youtube.com/watch?v=tavHJoum--g&list=PLRMa_gJ8vx... , thesis in question https://github.com/kastnerkyle/udem_masters_thesis https://github.com/kastnerkyle/udem_masters_thesis [5] Polyphonic Magenta model https://www.youtube.com/watch?v=s0BVFVqEY4A https://www.youtube.com/watch?v=s0BVFVqEY4A [6] RL tuning to improve generation https://magenta.tensorflow.org/2016/11/09/tuning-recurrent-networks-with-reinforcement-learning https://magenta.tensorflow.org/2016/11/09/tuning-recurrent-n... [7] Folk-RNN is similar but uses ABC representation (https://highnoongmt.wordpress.com/2015/05/22/lisls-stis-recurrent-neural-networks-for-folk-music-generation/ https://highnoongmt.wordpress.com/2015/05/22/lisls-stis-recu...), https://soundcloud.com/seaandsailor/sets/char-rnn-composes-irish-folk-music https://soundcloud.com/seaandsailor/sets/char-rnn-composes-i..., also see related post https://maraoz.com/2016/02/02/abc-rnn/ https://maraoz.com/2016/02/02/abc-rnn/ [8] DeepBach , http://www.flow-machines.com/deepbach-polyphonic-music-generation-bach-chorales/ http://www.flow-machines.com/deepbach-polyphonic-music-gener... [9] Lawson He's LSTM approach, http://www.lawsonhe.com/music.html http://www.lawsonhe.com/music.html [10] Daddy's Car https://www.youtube.com/watch?v=LSHZ_b05W7o https://www.youtube.com/watch?v=LSHZ_b05W7o [11] Interactive Magenta Jam https://magenta.tensorflow.org/2016/12/16/nips-demo https://magenta.tensorflow.org/2016/12/16/nips-demo [12] https://www.technologyreview.com/s/604010/google-brain-wants-creative-ai-to-help-humans-make-a-new-kind-of-art/ https://www.technologyreview.com/s/604010/google-brain-wants...

dougeck 9y ago
Wow. Better bib than we ourselves have! :)

deleted 9y ago

[deleted]

dougeck 9y ago

I've used the same analogy. Its unconditional and produces no coherent structure, but at least it captures some pianist-like timing and dynamics. Magenta is a github project ... please do something to help! :-)

6stringmerc 9y ago
Eh, asking a musician to write code is like asking a programmer to learn a scale on an accordion. Either you're going to see the failures in your own art, and be open to criticism, or ask for somebody to do it for you. I'll put it thusly: As a broke-ass musician barely making ends meet, I'll be more than happy to contribute if Alphabet pays off what's left of the mortgage on my house. Until then, I've only got the perspective that a project like this is trying to take food off my plate, money out of my pocket, and replace me. If you want my help, Son, you got to come better at me than with Altruism. Otherwise, the best thing I can do to help is to say "This project is a heap of shit and your time is better spent elsewhere" without tinging it with too much condescension. Your child is brain damaged. Take it to Oregon.