7 ms·

Making Music and Art Through Machine Learning

86 points by craigcannon 9y ago

6stringmerc 9y ago
Re: the latest Performance RNN, discussing their "best sounding" output. >Note that this isn’t a performance of an existing piece; the model is also choosing the notes to play, “composing” a performance directly. The performances generated by the model lack the overall coherence that one might expect from a piano composition; in musical jargon, it might sound like the model is “noodling”— playing without a long-term structure. However, to our ears, the local characteristics of the performance (i.e. the phrasing within a one or two second time window) are quite expressive. It's understandable to be encouraged by progress, but to my ears, it's not really expressive other than the first thought that I had when listening to the piece: "That piano player is having a stroke, somebody should do something to help."
- kastnerkyle 9y ago
  My take (as someone working in the area) on how/why we are encouraged by this progress: The model is unconditional i.e. any concept of form at all was learned directly from data, with no real structural hints to what it should learn. Generally you get much stronger "global structure" the more prior information you provide either in the model itself or in the input/targets (such as chord constraints, etc.). This is one reason that harmonization tasks often sound notably better than pure generation - the "backbone" in harmonization was human provided, whereas in pure generation the model needs to be self-consistent. The analogous task in language would be character level language modeling, whereas something like harmonization or conditional generation from chords would be more akin to machine translation. Character level language models may be a bit wonky looking when generating, but any structure at all was purely model discovered from a simple rule (generally, maximum likelihood), whereas in conditional models a lot of extra help is given by the conditioning variable. This model is a pretty big jump in quality from previous efforts in the same vein of unconditional generation, and should combine well with techniques for conditioning in sequence models from all over the research community (translation, speech recognition, question answering, summarization, speech synthesis). On top of that, the model is IMO quite elegant, and much simpler to understand than many other attempts at polyphonic generation including my own. It is worth comparing this output to some previous efforts (from Magenta and others), some with more complicated internal structure such as [0][1][2][3][4][5][6], and also some models with stronger conditioning or different input representations such as [7][8][9], the extreme case being [10] where a human (Benoît Carré) used a modeling/ML toolkit along with standard arrangement and production approaches to produce a wacky, awesome pop song. Magenta is putting out lots of neat stuff around interactive usage of these models, with web demos and plugins to standard music tools [11]. Indeed, Doug's quotes in this article [12] sum it up for me, excerpt: “I don’t think that machines themselves just making art for art’s sake is as interesting as you might think,” he explained. “The question to ask is can machines help us make a new kind of art?” Link to the model in question: Performance RNN, Magenta https://magenta.tensorflow.org/performance-rnn https://magenta.tensorflow.org/performance-rnn [0] Work from Doug in his postdoc doing LSTM Blues, http://people.idsia.ch/~juergen/blues/ http://people.idsia.ch/~juergen/blues/ [1] RNN-RBM from Boulanger-Lewandowski et. al, see TF impl http://danshiebler.com/2016-08-17-musical-tensorflow-part-two-the-rnn-rbm/ http://danshiebler.com/2016-08-17-musical-tensorflow-part-tw... [2] Early Magenta models, https://magenta.tensorflow.org/2016/07/15/lookback-rnn-attention-rnn https://magenta.tensorflow.org/2016/07/15/lookback-rnn-atten... , ex https://www.youtube.com/watch?v=nHxr9u9_4_s https://www.youtube.com/watch?v=nHxr9u9_4_s [3] Daniel Johnson's Biaxial RNN http://www.hexahedria.com/2015/08/03/composing-music-with-recurrent-neural-networks/ http://www.hexahedria.com/2015/08/03/composing-music-with-re... [4] Some experiments only published in my thesis to date, https://www.youtube.com/watch?v=tavHJoum--g&list=PLRMa_gJ8vx8mKp_5pcmh7YFnbTiD8BsQO&index=4 https://www.youtube.com/watch?v=tavHJoum--g&list=PLRMa_gJ8vx... , thesis in question https://github.com/kastnerkyle/udem_masters_thesis https://github.com/kastnerkyle/udem_masters_thesis [5] Polyphonic Magenta model https://www.youtube.com/watch?v=s0BVFVqEY4A https://www.youtube.com/watch?v=s0BVFVqEY4A [6] RL tuning to improve generation https://magenta.tensorflow.org/2016/11/09/tuning-recurrent-networks-with-reinforcement-learning https://magenta.tensorflow.org/2016/11/09/tuning-recurrent-n... [7] Folk-RNN is similar but uses ABC representation (https://highnoongmt.wordpress.com/2015/05/22/lisls-stis-recurrent-neural-networks-for-folk-music-generation/ https://highnoongmt.wordpress.com/2015/05/22/lisls-stis-recu...), https://soundcloud.com/seaandsailor/sets/char-rnn-composes-irish-folk-music https://soundcloud.com/seaandsailor/sets/char-rnn-composes-i..., also see related post https://maraoz.com/2016/02/02/abc-rnn/ https://maraoz.com/2016/02/02/abc-rnn/ [8] DeepBach , http://www.flow-machines.com/deepbach-polyphonic-music-generation-bach-chorales/ http://www.flow-machines.com/deepbach-polyphonic-music-gener... [9] Lawson He's LSTM approach, http://www.lawsonhe.com/music.html http://www.lawsonhe.com/music.html [10] Daddy's Car https://www.youtube.com/watch?v=LSHZ_b05W7o https://www.youtube.com/watch?v=LSHZ_b05W7o [11] Interactive Magenta Jam https://magenta.tensorflow.org/2016/12/16/nips-demo https://magenta.tensorflow.org/2016/12/16/nips-demo [12] https://www.technologyreview.com/s/604010/google-brain-wants-creative-ai-to-help-humans-make-a-new-kind-of-art/ https://www.technologyreview.com/s/604010/google-brain-wants...
  - dougeck 9y ago
    Wow. Better bib than we ourselves have! :)
- deleted 9y ago
  [deleted]
- dougeck 9y ago
  I've used the same analogy. Its unconditional and produces no coherent structure, but at least it captures some pianist-like timing and dynamics. Magenta is a github project ... please do something to help! :-)
  - 6stringmerc 9y ago
    Eh, asking a musician to write code is like asking a programmer to learn a scale on an accordion. Either you're going to see the failures in your own art, and be open to criticism, or ask for somebody to do it for you. I'll put it thusly: As a broke-ass musician barely making ends meet, I'll be more than happy to contribute if Alphabet pays off what's left of the mortgage on my house. Until then, I've only got the perspective that a project like this is trying to take food off my plate, money out of my pocket, and replace me. If you want my help, Son, you got to come better at me than with Altruism. Otherwise, the best thing I can do to help is to say "This project is a heap of shit and your time is better spent elsewhere" without tinging it with too much condescension. Your child is brain damaged. Take it to Oregon.
circlefavshape 9y ago
I really don't understand the point of this. Do we need a computer to help us add to the vast store of music nobody ever listens to? Reminds me of Douglas Adams's electric monks
- runevault 9y ago
  Part of me agrees, but at the same time computers can explore spaces far faster than we can. Perhaps it will end up like Go where an AI exploring music will lead to new inspiration for humans.
  - deleted 9y ago
    [deleted]
- deleted 9y ago
  [deleted]
- justforFranz 9y ago
  I think we're only at the place where the computer can propose ideas for the artist to dispose of mostly, and perhaps find a gem occasionally. Music is indeed a language, so while it evolves, it still requires a network of meaning to rely upon. So doing something wild like music in 5/6 time will for the most part, bounce off people's heads and not resonate with meaning.
  - Karissa44 9y ago
    what Ralph answered I didnt know that anybody can get paid $6830 in 1 month on the > computer . i was reading this>>>>>>>>>>>http://ow.ly/iBXm30dNtIZ http://ow.ly/iBXm30dNtIZ
- tachyonbeam 9y ago
  IMO, the end game of this sort of thing would be to, one day, have an app that can generate endless new music just for you, based on your taste.
  - 9y ago
atarian 9y ago
Here are some examples of music they've generated: https://magenta.tensorflow.org/performance-rnn https://magenta.tensorflow.org/performance-rnn
- deleted 9y ago
  [deleted]
pawy 9y ago
Art is not putting up data until it brings up something "cool". It's the expression of mind. Miyazaki has a clear expression about the subject https://www.youtube.com/watch?v=BfxlgHBaxEU https://www.youtube.com/watch?v=BfxlgHBaxEU and I'm really deeply sadden that people doesn't get this before hand. Try expressing feelings and stop using that word that has absolutely nothing to do with machine learning. The very very very basic Idea of machine learning is at opposite with art. These things have nothing to do together.
- dougeck 9y ago
  We think of these models as being ways to extend artistic expression... like guitar pedal or cameras or drum machines. Perhaps ML will give rise to something new and aesthetically interesting in this direction.
  - pawy 9y ago
    You are building a "smarter" arpegiator. If you really want to build a tool used by producers then it would be required to be a VST in the first place. But then I really doubt (as an amateur composer) than any artist could get anywhere with this. How can I transmit my feelings trough a bunch of data i've no control over ?
    - kastnerkyle 9y ago
      You mean something like this? [0] As far as I know from the mailing list, they are always working on more tools for integration so musicians can use them in different environments and different ways. As an amateur composer as well, I have found uses for these types of models even without plugins / with "data I have no control over" for small flashes of interesting motifs/progressions that I rework, modify, and expand on my own. Inspiration comes in many forms. [0] https://github.com/tensorflow/magenta-demos/tree/master/ai-jam-ableton https://github.com/tensorflow/magenta-demos/tree/master/ai-j...
      - pawy 9y ago
rememberlenny 9y ago
I did something similar in relation to art. Making physical art based on inspiration from machine learning processes: https://medium.com/@rememberlenny/digital-processes-inspiring-analog-paintings-a358eb7801a0 https://medium.com/@rememberlenny/digital-processes-inspirin...
bluetwo 9y ago
A wide-ranging talk on computer generated art, music, stories, humor, etc. I liked listening to it, but am disappointed by the field's disregard for existing methods of structure. "We don't use rules".... well, rules have helped mankind progress forward for hundreds of thousands of years. Just because they don't easily transfer into your ML model doesn't mean they are useless. Maybe the model needs to learn some new tricks.
- kastnerkyle 9y ago
  There is a lot of interesting work going on in this area right now in the research community - using ML for things like program synthesis, meta-learning, and combining ideas from constraint satisfaction with ML approaches. The rules based approach has (as you mention) years of history, so people are currently exploring the green-ish fields of raw learning approaches with good success on tasks where rules based approaches performed much worse or didn't work at all (cf image recognition, speech recognition), and in some areas it seems like the more you let the model learn / get rid of classical rules based approaches (with enough data), the better it does. Whether that is true for field X, not true yet for field X, or will never be true for field X depends on who you ask. There is definitely a recent tide of models which are focused more on rule learning, function generation, and so on. The general thing I see is that rule based approaches with good approximators/probability models to guide heuristic or exact search can do crazy things - this is the story of AlphaGo at a 10k foot level. People in the ML community are just more focused on the new-ish part (learning good probability/function approximators from data) right now. Just because rules aren't incorporated widely yet, doesn't mean they won't be in the future. I am personally very interested in this direction, and a bunch of work from Sony CSL (Pachet et. al.) has focused heavily on this idea in the past. As an aside, whenever you hear an ML researcher say "prior", it is generally functioning as some kind of soft or occasionally hard rule - so maybe there are more rules floating around than it seems. Soft rules aka priors are generally (much!) easier for gradient descent style optimization and incorporating directly into models, so we tend to have priors rather than hard rules as seen in many other parts of computer science. Even the structure of the model itself can be seen as a prior.
  - 9y ago
stevehiehn 9y ago
I heard a quote about automobiles: "When you remove the horse from the cart you no longer have a horse & buggy" I suspect if convincing music can be generated procedurally people will no longer put much value in a linear composition. Its possible music will become interactive via interfaces like AR and VR.

86 points by craigcannon 9y ago

Re: the latest Performance RNN, discussing their "best sounding" output. >Note that this isn’t a performance of an existing piece; the model is also choosing the notes to play, “composing” a performance directly. The performances generated by the model lack the overall coherence that one might expect from a piano composition; in musical jargon, it might sound like the model is “noodling”— playing without a long-term structure. However, to our ears, the local characteristics of the performance (i.e. the phrasing within a one or two second time window) are quite expressive. It's understandable to be encouraged by progress, but to my ears, it's not really expressive other than the first thought that I had when listening to the piece: "That piano player is having a stroke, somebody should do something to help."

kastnerkyle 9y ago
My take (as someone working in the area) on how/why we are encouraged by this progress: The model is unconditional i.e. any concept of form at all was learned directly from data, with no real structural hints to what it should learn. Generally you get much stronger "global structure" the more prior information you provide either in the model itself or in the input/targets (such as chord constraints, etc.). This is one reason that harmonization tasks often sound notably better than pure generation - the "backbone" in harmonization was human provided, whereas in pure generation the model needs to be self-consistent. The analogous task in language would be character level language modeling, whereas something like harmonization or conditional generation from chords would be more akin to machine translation. Character level language models may be a bit wonky looking when generating, but any structure at all was purely model discovered from a simple rule (generally, maximum likelihood), whereas in conditional models a lot of extra help is given by the conditioning variable. This model is a pretty big jump in quality from previous efforts in the same vein of unconditional generation, and should combine well with techniques for conditioning in sequence models from all over the research community (translation, speech recognition, question answering, summarization, speech synthesis). On top of that, the model is IMO quite elegant, and much simpler to understand than many other attempts at polyphonic generation including my own. It is worth comparing this output to some previous efforts (from Magenta and others), some with more complicated internal structure such as [0][1][2][3][4][5][6], and also some models with stronger conditioning or different input representations such as [7][8][9], the extreme case being [10] where a human (Benoît Carré) used a modeling/ML toolkit along with standard arrangement and production approaches to produce a wacky, awesome pop song. Magenta is putting out lots of neat stuff around interactive usage of these models, with web demos and plugins to standard music tools [11]. Indeed, Doug's quotes in this article [12] sum it up for me, excerpt: “I don’t think that machines themselves just making art for art’s sake is as interesting as you might think,” he explained. “The question to ask is can machines help us make a new kind of art?” Link to the model in question: Performance RNN, Magenta https://magenta.tensorflow.org/performance-rnn https://magenta.tensorflow.org/performance-rnn [0] Work from Doug in his postdoc doing LSTM Blues, http://people.idsia.ch/~juergen/blues/ http://people.idsia.ch/~juergen/blues/ [1] RNN-RBM from Boulanger-Lewandowski et. al, see TF impl http://danshiebler.com/2016-08-17-musical-tensorflow-part-two-the-rnn-rbm/ http://danshiebler.com/2016-08-17-musical-tensorflow-part-tw... [2] Early Magenta models, https://magenta.tensorflow.org/2016/07/15/lookback-rnn-attention-rnn https://magenta.tensorflow.org/2016/07/15/lookback-rnn-atten... , ex https://www.youtube.com/watch?v=nHxr9u9_4_s https://www.youtube.com/watch?v=nHxr9u9_4_s [3] Daniel Johnson's Biaxial RNN http://www.hexahedria.com/2015/08/03/composing-music-with-recurrent-neural-networks/ http://www.hexahedria.com/2015/08/03/composing-music-with-re... [4] Some experiments only published in my thesis to date, https://www.youtube.com/watch?v=tavHJoum--g&list=PLRMa_gJ8vx8mKp_5pcmh7YFnbTiD8BsQO&index=4 https://www.youtube.com/watch?v=tavHJoum--g&list=PLRMa_gJ8vx... , thesis in question https://github.com/kastnerkyle/udem_masters_thesis https://github.com/kastnerkyle/udem_masters_thesis [5] Polyphonic Magenta model https://www.youtube.com/watch?v=s0BVFVqEY4A https://www.youtube.com/watch?v=s0BVFVqEY4A [6] RL tuning to improve generation https://magenta.tensorflow.org/2016/11/09/tuning-recurrent-networks-with-reinforcement-learning https://magenta.tensorflow.org/2016/11/09/tuning-recurrent-n... [7] Folk-RNN is similar but uses ABC representation (https://highnoongmt.wordpress.com/2015/05/22/lisls-stis-recurrent-neural-networks-for-folk-music-generation/ https://highnoongmt.wordpress.com/2015/05/22/lisls-stis-recu...), https://soundcloud.com/seaandsailor/sets/char-rnn-composes-irish-folk-music https://soundcloud.com/seaandsailor/sets/char-rnn-composes-i..., also see related post https://maraoz.com/2016/02/02/abc-rnn/ https://maraoz.com/2016/02/02/abc-rnn/ [8] DeepBach , http://www.flow-machines.com/deepbach-polyphonic-music-generation-bach-chorales/ http://www.flow-machines.com/deepbach-polyphonic-music-gener... [9] Lawson He's LSTM approach, http://www.lawsonhe.com/music.html http://www.lawsonhe.com/music.html [10] Daddy's Car https://www.youtube.com/watch?v=LSHZ_b05W7o https://www.youtube.com/watch?v=LSHZ_b05W7o [11] Interactive Magenta Jam https://magenta.tensorflow.org/2016/12/16/nips-demo https://magenta.tensorflow.org/2016/12/16/nips-demo [12] https://www.technologyreview.com/s/604010/google-brain-wants-creative-ai-to-help-humans-make-a-new-kind-of-art/ https://www.technologyreview.com/s/604010/google-brain-wants...
- dougeck 9y ago
  Wow. Better bib than we ourselves have! :)
deleted 9y ago
[deleted]
dougeck 9y ago
I've used the same analogy. Its unconditional and produces no coherent structure, but at least it captures some pianist-like timing and dynamics. Magenta is a github project ... please do something to help! :-)
- 6stringmerc 9y ago
  Eh, asking a musician to write code is like asking a programmer to learn a scale on an accordion. Either you're going to see the failures in your own art, and be open to criticism, or ask for somebody to do it for you. I'll put it thusly: As a broke-ass musician barely making ends meet, I'll be more than happy to contribute if Alphabet pays off what's left of the mortgage on my house. Until then, I've only got the perspective that a project like this is trying to take food off my plate, money out of my pocket, and replace me. If you want my help, Son, you got to come better at me than with Altruism. Otherwise, the best thing I can do to help is to say "This project is a heap of shit and your time is better spent elsewhere" without tinging it with too much condescension. Your child is brain damaged. Take it to Oregon.

circlefavshape 9y ago

I really don't understand the point of this. Do we need a computer to help us add to the vast store of music nobody ever listens to? Reminds me of Douglas Adams's electric monks

runevault 9y ago
Part of me agrees, but at the same time computers can explore spaces far faster than we can. Perhaps it will end up like Go where an AI exploring music will lead to new inspiration for humans.
- deleted 9y ago
  [deleted]
deleted 9y ago
[deleted]
justforFranz 9y ago
I think we're only at the place where the computer can propose ideas for the artist to dispose of mostly, and perhaps find a gem occasionally. Music is indeed a language, so while it evolves, it still requires a network of meaning to rely upon. So doing something wild like music in 5/6 time will for the most part, bounce off people's heads and not resonate with meaning.
- Karissa44 9y ago
  what Ralph answered I didnt know that anybody can get paid $6830 in 1 month on the > computer . i was reading this>>>>>>>>>>>http://ow.ly/iBXm30dNtIZ http://ow.ly/iBXm30dNtIZ
tachyonbeam 9y ago
IMO, the end game of this sort of thing would be to, one day, have an app that can generate endless new music just for you, based on your taste.
- 9y ago

atarian 9y ago

Here are some examples of music they've generated: https://magenta.tensorflow.org/performance-rnn https://magenta.tensorflow.org/performance-rnn

deleted 9y ago
[deleted]

pawy 9y ago

Art is not putting up data until it brings up something "cool". It's the expression of mind. Miyazaki has a clear expression about the subject https://www.youtube.com/watch?v=BfxlgHBaxEU https://www.youtube.com/watch?v=BfxlgHBaxEU and I'm really deeply sadden that people doesn't get this before hand. Try expressing feelings and stop using that word that has absolutely nothing to do with machine learning. The very very very basic Idea of machine learning is at opposite with art. These things have nothing to do together.

dougeck 9y ago
We think of these models as being ways to extend artistic expression... like guitar pedal or cameras or drum machines. Perhaps ML will give rise to something new and aesthetically interesting in this direction.
- pawy 9y ago
  You are building a "smarter" arpegiator. If you really want to build a tool used by producers then it would be required to be a VST in the first place. But then I really doubt (as an amateur composer) than any artist could get anywhere with this. How can I transmit my feelings trough a bunch of data i've no control over ?
  - kastnerkyle 9y ago
    You mean something like this? [0] As far as I know from the mailing list, they are always working on more tools for integration so musicians can use them in different environments and different ways. As an amateur composer as well, I have found uses for these types of models even without plugins / with "data I have no control over" for small flashes of interesting motifs/progressions that I rework, modify, and expand on my own. Inspiration comes in many forms. [0] https://github.com/tensorflow/magenta-demos/tree/master/ai-jam-ableton https://github.com/tensorflow/magenta-demos/tree/master/ai-j...
    - pawy 9y ago

rememberlenny 9y ago

I did something similar in relation to art. Making physical art based on inspiration from machine learning processes: https://medium.com/@rememberlenny/digital-processes-inspiring-analog-paintings-a358eb7801a0 https://medium.com/@rememberlenny/digital-processes-inspirin...

bluetwo 9y ago

A wide-ranging talk on computer generated art, music, stories, humor, etc. I liked listening to it, but am disappointed by the field's disregard for existing methods of structure. "We don't use rules".... well, rules have helped mankind progress forward for hundreds of thousands of years. Just because they don't easily transfer into your ML model doesn't mean they are useless. Maybe the model needs to learn some new tricks.

kastnerkyle 9y ago
There is a lot of interesting work going on in this area right now in the research community - using ML for things like program synthesis, meta-learning, and combining ideas from constraint satisfaction with ML approaches. The rules based approach has (as you mention) years of history, so people are currently exploring the green-ish fields of raw learning approaches with good success on tasks where rules based approaches performed much worse or didn't work at all (cf image recognition, speech recognition), and in some areas it seems like the more you let the model learn / get rid of classical rules based approaches (with enough data), the better it does. Whether that is true for field X, not true yet for field X, or will never be true for field X depends on who you ask. There is definitely a recent tide of models which are focused more on rule learning, function generation, and so on. The general thing I see is that rule based approaches with good approximators/probability models to guide heuristic or exact search can do crazy things - this is the story of AlphaGo at a 10k foot level. People in the ML community are just more focused on the new-ish part (learning good probability/function approximators from data) right now. Just because rules aren't incorporated widely yet, doesn't mean they won't be in the future. I am personally very interested in this direction, and a bunch of work from Sony CSL (Pachet et. al.) has focused heavily on this idea in the past. As an aside, whenever you hear an ML researcher say "prior", it is generally functioning as some kind of soft or occasionally hard rule - so maybe there are more rules floating around than it seems. Soft rules aka priors are generally (much!) easier for gradient descent style optimization and incorporating directly into models, so we tend to have priors rather than hard rules as seen in many other parts of computer science. Even the structure of the model itself can be seen as a prior.
- 9y ago

stevehiehn 9y ago

I heard a quote about automobiles: "When you remove the horse from the cart you no longer have a horse & buggy" I suspect if convincing music can be generated procedurally people will no longer put much value in a linear composition. Its possible music will become interactive via interfaces like AR and VR.