6 ms·

Algorithmic Music Generation With Recurrent Neural Networks [video]

67 points by rndn 11y ago

m-i-l 11y ago
Not much information in the video on how this was achieved, but a quick search for "gruv algorithmic music generation" returns the following: https://sites.google.com/site/anayebihomepage/cs224dfinalproject https://sites.google.com/site/anayebihomepage/cs224dfinalpro... . Extract: We compare the performance of two different types of recurrent neural networks (RNNs) for the task of algorithmic music generation, with audio waveforms as input (as opposed to the standard MIDI). In particular, we focus on RNNs that have a sophisticated gating mechanism, namely, the Long Short-Term Memory (LSTM) network and the recently introduced Gated Recurrent Unit (GRU). Our results indicate that the generated outputs of the LSTM network were significantly more musically plausible than those of the GRU.
crucialfelix 11y ago
The funny thing is that the only really good ones are the first few where they claim its just random noise. The later ones just sound like a crappy radio. With images the technique works because we like looking at the dense artifacts. Millions of dog heads essentially copy and pasted onto any appendage that looks like it should be a head. It looks like drugs and overwhelms in the same way. If you just take white noise and throw it through a tuned filter bank (which is in essence and in final effect what they are doing here) then you just get crappy audio. The more standard and successful use of NN in composition is the use it on pitch series and compositional forms. Like feeding it all of Beethoven and then getting it to generate similar compositions. That's been going on for decades. You can do it with that kind of data. But the thing about pop and electronic music is that the easily machine observable elements are not very interesting. Listen to the 4/4 kick and snare pattern in the video. Its boring as hell. (Other tracks can be just a kick and snare and they are amazing and we celebrate them as classics and play them for 20 years. Machines will never understand why) What's great and essential are things like spatial relationships between elements in the mix: how does the surge of the compressed synth/guitar cause the beat to tumble outward and stir you up ? after a series of peaks in a synth melody then the next time it pulls back creating a space that pulls at your heart strings. you create a negative space that the listener goes into. playing with listeners expectations based on what songs, conventions and tropes they already know and respond to.
- sweezyjeezy 11y ago
  The image stuff works because we have found a way to model a good prior for it : convolution layers are basically enforcing some positional invariance and locality constraints on what our model believes the world looks like. Without this very strict prior, image recognition with neural networks just wouldn't really work. We haven't found a way to enforce a good prior for temporal data like sound yet.
  - TheOtherHobbes 11y ago
    I realised a while back a lot of computer music is really computer science music - it's people who know something about computers but not much about the art of music, playing with relatively trivial algorithms to create music-like results. There's also the academic musical equivalent - music professors using stock faddy techniques like serialism or (currently) number and group theory. It's not that this is an impossible problem. It's more that the set of people who can code machine learning algorithms and understand music theory and are creative enough to invent new algorithmic techniques and to create more-than-listenable music is incredibly small - double figures, if that. So progress in non-trivial computer music has been incredibly slow. The DSP side has been far more successful, because DSP is - in most ways - a much simpler problem.
    - crucialfelix 11y ago
      The reason for it has a lot to do with how Academia works. You need to write papers and produce innovative works. If you don't then you won't get funding or advance your career. > or (currently) number and group theory. That was Xenakis back in the 1950s ! Xenakis is the exception. He's really the foundation of academic computer music and his music is amazing and moving and his compositional concepts are still being hacked out by music programmers today.
    - skwosh 11y ago
      I somewhat agree about computer music, though it appears to be an extension of the process driven composition that has been part of western (art) music for a while now (e.g. modulation). Sometimes I wonder if linguistics has more to offer composition than algebra (speculation, as I know next to nothing about (non-CS) linguistics). > It's not that this is an impossible problem. It's more that the set of people who can code machine learning algorithms and understand music theory and are creative enough to invent new algorithmic techniques and to create more-than-listenable music is incredibly small - double figures, if that. Are you pursuing something like this? Or know anyone who is? This is one of my main interests (alongside better interfaces for composition and well, making music). I'm actually back at University for my second degree to study this sort of thing. If you have a blog or anything I'd be interested to find out...
msamwald 11y ago
It would be interesting to know how novel those sequences are (obviously, the outcome would be far less impressive if what we hear is basically a looped, noisy sample of a song that already exists).
acd 11y ago
I think it would sound better if we thought the neural network to play notes and music theory.
leaveyou 11y ago
Another promising field is RNN applied to TED talks: youtube.com/watch?v=-OodHtJ1saY
mpdehaan2 11y ago
I'm not 100% up to speed on my AI, but this sounds about like what you'd get with random variations on a signal, where the neural net is the "which sounds like X" filter, and picks one of the two to survive. But that would be using both some form of a genetic algorithm (details TBD?) and the neural net as the checker. But is it? If they aren't doing it that way, I'd be interested in hearing how it's evolving the signal in that given direction - and also how that filter works (what libraries does it use?). Sounds like it hit some sort of local maxima, so this system won't ever produce the original song, but something a percentage of the way toward it. I'm a bit more interested in algorithmic composition, but this could be interesting if trying to blend genres. For a long time I've wanted to build a program that could produce essentially an infinite song morphing between genres with lots of tunable parameters.
anentropic 11y ago
This is not music. Music is not simply organised sound. Music is a cultural practise.
- yyyyes 11y ago
  This is not a cultural practice?
durbin 11y ago
Source code link?

67 points by rndn 11y ago

m-i-l 11y ago

Not much information in the video on how this was achieved, but a quick search for "gruv algorithmic music generation" returns the following: https://sites.google.com/site/anayebihomepage/cs224dfinalproject https://sites.google.com/site/anayebihomepage/cs224dfinalpro... . Extract: We compare the performance of two different types of recurrent neural networks (RNNs) for the task of algorithmic music generation, with audio waveforms as input (as opposed to the standard MIDI). In particular, we focus on RNNs that have a sophisticated gating mechanism, namely, the Long Short-Term Memory (LSTM) network and the recently introduced Gated Recurrent Unit (GRU). Our results indicate that the generated outputs of the LSTM network were significantly more musically plausible than those of the GRU.

crucialfelix 11y ago

The funny thing is that the only really good ones are the first few where they claim its just random noise. The later ones just sound like a crappy radio. With images the technique works because we like looking at the dense artifacts. Millions of dog heads essentially copy and pasted onto any appendage that looks like it should be a head. It looks like drugs and overwhelms in the same way. If you just take white noise and throw it through a tuned filter bank (which is in essence and in final effect what they are doing here) then you just get crappy audio. The more standard and successful use of NN in composition is the use it on pitch series and compositional forms. Like feeding it all of Beethoven and then getting it to generate similar compositions. That's been going on for decades. You can do it with that kind of data. But the thing about pop and electronic music is that the easily machine observable elements are not very interesting. Listen to the 4/4 kick and snare pattern in the video. Its boring as hell. (Other tracks can be just a kick and snare and they are amazing and we celebrate them as classics and play them for 20 years. Machines will never understand why) What's great and essential are things like spatial relationships between elements in the mix: how does the surge of the compressed synth/guitar cause the beat to tumble outward and stir you up ? after a series of peaks in a synth melody then the next time it pulls back creating a space that pulls at your heart strings. you create a negative space that the listener goes into. playing with listeners expectations based on what songs, conventions and tropes they already know and respond to.

sweezyjeezy 11y ago
The image stuff works because we have found a way to model a good prior for it : convolution layers are basically enforcing some positional invariance and locality constraints on what our model believes the world looks like. Without this very strict prior, image recognition with neural networks just wouldn't really work. We haven't found a way to enforce a good prior for temporal data like sound yet.
- TheOtherHobbes 11y ago
  I realised a while back a lot of computer music is really computer science music - it's people who know something about computers but not much about the art of music, playing with relatively trivial algorithms to create music-like results. There's also the academic musical equivalent - music professors using stock faddy techniques like serialism or (currently) number and group theory. It's not that this is an impossible problem. It's more that the set of people who can code machine learning algorithms and understand music theory and are creative enough to invent new algorithmic techniques and to create more-than-listenable music is incredibly small - double figures, if that. So progress in non-trivial computer music has been incredibly slow. The DSP side has been far more successful, because DSP is - in most ways - a much simpler problem.
  - crucialfelix 11y ago
    The reason for it has a lot to do with how Academia works. You need to write papers and produce innovative works. If you don't then you won't get funding or advance your career. > or (currently) number and group theory. That was Xenakis back in the 1950s ! Xenakis is the exception. He's really the foundation of academic computer music and his music is amazing and moving and his compositional concepts are still being hacked out by music programmers today.
  - skwosh 11y ago
    I somewhat agree about computer music, though it appears to be an extension of the process driven composition that has been part of western (art) music for a while now (e.g. modulation). Sometimes I wonder if linguistics has more to offer composition than algebra (speculation, as I know next to nothing about (non-CS) linguistics). > It's not that this is an impossible problem. It's more that the set of people who can code machine learning algorithms and understand music theory and are creative enough to invent new algorithmic techniques and to create more-than-listenable music is incredibly small - double figures, if that. Are you pursuing something like this? Or know anyone who is? This is one of my main interests (alongside better interfaces for composition and well, making music). I'm actually back at University for my second degree to study this sort of thing. If you have a blog or anything I'd be interested to find out...

msamwald 11y ago

It would be interesting to know how novel those sequences are (obviously, the outcome would be far less impressive if what we hear is basically a looped, noisy sample of a song that already exists).

acd 11y ago

I think it would sound better if we thought the neural network to play notes and music theory.

leaveyou 11y ago

Another promising field is RNN applied to TED talks: youtube.com/watch?v=-OodHtJ1saY

mpdehaan2 11y ago

I'm not 100% up to speed on my AI, but this sounds about like what you'd get with random variations on a signal, where the neural net is the "which sounds like X" filter, and picks one of the two to survive. But that would be using both some form of a genetic algorithm (details TBD?) and the neural net as the checker. But is it? If they aren't doing it that way, I'd be interested in hearing how it's evolving the signal in that given direction - and also how that filter works (what libraries does it use?). Sounds like it hit some sort of local maxima, so this system won't ever produce the original song, but something a percentage of the way toward it. I'm a bit more interested in algorithmic composition, but this could be interesting if trying to blend genres. For a long time I've wanted to build a program that could produce essentially an infinite song morphing between genres with lots of tunable parameters.

anentropic 11y ago

This is not music. Music is not simply organised sound. Music is a cultural practise.

yyyyes 11y ago
This is not a cultural practice?

durbin 11y ago

Source code link?