4 ms·

Show HN: Catchy melodies made with a diffusion-based neural net assistant

38 points by zone411 4y ago

I've created a diffusion-based neural net generative assistant that makes creating new melodies much easier, even for non-musicians like me. These are meant to be just the catchy "hook" parts of songs, so more work is required to make them into full songs, but this is already handled well by existing products (e.g. there are plugins that can suggest a few possible chord progressions based on the melody and there is even good singing software that I used without any tweaks to make the “voice” playlist: Synthesizer V Studio). This side project turned out to be quite challenging because of how little data there is to train on - several orders of magnitude less than DALL-E or GPT-3 had available for its training, so it required a deep dive into research of new generalization and augmentation techniques and some feature engineering. Various other instruments: Voice: https://www.youtube.com/playlist?list=PLoCzMRqh5SkE1yC8_WtJ-WtZwySFw23UX https://www.youtube.com/playlist?list=PLoCzMRqh5SkE1yC8_WtJ-... Synth: https://www.youtube.com/playlist?list=PLoCzMRqh5SkFj7RNZvjr7PRw_8NR7fN1R https://www.youtube.com/playlist?list=PLoCzMRqh5SkFj7RNZvjr7... Bell: https://www.youtube.com/playlist?list=PLoCzMRqh5SkEYHYvHX9m9nlNdMHzbJJhC https://www.youtube.com/playlist?list=PLoCzMRqh5SkEYHYvHX9m9... Guitar: https://www.youtube.com/playlist?list=PLoCzMRqh5SkGKvfkP2OexojW7XGiuDl5L https://www.youtube.com/playlist?list=PLoCzMRqh5SkGKvfkP2Oex... Sax: https://www.youtube.com/playlist?list=PLoCzMRqh5SkHfsZgzzdSh-mrNFe4qLCYt https://www.youtube.com/playlist?list=PLoCzMRqh5SkHfsZgzzdSh... Grand Piano: https://www.youtube.com/playlist?list=PLoCzMRqh5SkFMch5x60uhBxv-aVH70VOX https://www.youtube.com/playlist?list=PLoCzMRqh5SkFMch5x60uh... SoundCloud electric piano: https://soundcloud.com/lech-mazur-995769534/sets/ai-assistant-melodies-electric-piano?utm_source=clipboard&utm_medium=text&utm_campaign=social_sharing https://soundcloud.com/lech-mazur-995769534/sets/ai-assistan... SoundCloud vocal: https://soundcloud.com/lech-mazur-995769534/sets/ai-assistant-melodies-voice?utm_source=clipboard&utm_medium=text&utm_campaign=social_sharing https://soundcloud.com/lech-mazur-995769534/sets/ai-assistan...

mcphage 4y ago
After listening to the same pieces over and over again on my kids' music boxes and whatnot, I've really wanted a music box that just auto-generated new 0:15 second pieces every time instead of a static loop.
deleted 4y ago
[deleted]
stolenmerch 4y ago
This is really neat. Is there a colab or Jupyter notebook we can look at?
- zone411 4y ago
  Not yet. The current version is not fully automated and it requires human input, so it will take a decent amount of work and documentation to make it ready for users. It also mixes C++ with Python in a few places for performance reasons when generating melodies and I don't know if that's even possible on Colab?
cuttysnark 4y ago
This is fascinating. I've always loved the idea of a .midi melody being able to be mapped to a "vocal" track and transformed accordingly—akin to Autotune The News, but with AI/ML instead of by-hand. In a sense, the lyrics could be spoken and then applied to the .midi melody. The result would sound similar to the SoundCloud vocal OP link, but with "words".
- zone411 4y ago
  > In a sense, the lyrics could be spoken and then applied to the .midi melody. The result would sound similar to the SoundCloud vocal OP link, but with "words". Yes, Synthesizer V Studio can do it quite well! I've added some silly lyrics to a few of these melodies and I was quite happy with the results. I'll upload them to the same YouTube channel. Here is a cover song that somebody made: https://www.youtube.com/watch?v=ettSVy4K_DY https://www.youtube.com/watch?v=ettSVy4K_DY.
armchairhacker 4y ago
I think AI in music generation will end up being a big thing like with DALL-E for images and GPT3 for text. Possibly more because it seems like people have less intuition for creating music than they do for images and words.
electric_muse 4y ago
Nice work. Somehow this reminds me of Harvest Moon 64 music. I'm not sure I felt these were necessarily catchy (but I think that comes from repetition), but they were incredibly pleasant and interesting. Also, obligatory re: earworms: https://www.youtube.com/watch?v=iDjX4-LKqCA https://www.youtube.com/watch?v=iDjX4-LKqCA
p1esk 4y ago
Are you familiar with aiva.ai? They also use midi format. Though I thought after Jukebox everyone would switch to raw audio.
- chaosprint 4y ago
  https://github.com/chaosprint/RaveForce https://github.com/chaosprint/RaveForce How about both symbolic and subsymbolic?
- zone411 4y ago
  Yes, I am. It does many things well but last time I used it (about a year ago), I couldn't get it to do what I focused on with this project: create a catchy melodic hook that could be a base of a hit song or an earworm commercial jingle. I believe this requires an element of surprise in a melody - it's much easier to just create something pleasant-sounding, without anything fancy related to neural nets.
kastnerkyle 4y ago
I've had pretty good luck recently with a mix of SUNDAE (https://arxiv.org/abs/2112.06749 https://arxiv.org/abs/2112.06749) and coconet (https://arxiv.org/abs/1903.07227 https://arxiv.org/abs/1903.07227) and/or Music Transformer based internal models recently for modeling very small datasets of polyphonic "midified" music. Research paper hopefully soon to come... Not sure what your pipeline looks like, but those papers might be worth putting on your radar. And as you mention, symbolic music datasets are both surprisingly small, surprisingly low quality, and generally a huge pain to work with. Cool stuff - I like the sax! For anyone unfamiliar with diffusion models (and coconet / OrderlessNADE), one of the really nice properties of them as opposed to "standard" autoregressive (GPT / RNN) style models, is that you should be able to specify any part, and fill in any other part - rather than being forced to specify the "past" and predict only the "future". The coconet "doodle" is a good example of this interface at work (https://www.google.com/doodles/celebrating-johann-sebastian-bach https://www.google.com/doodles/celebrating-johann-sebastian-...) XLNet had some of this promise too (https://arxiv.org/abs/1906.08237 https://arxiv.org/abs/1906.08237) but I never had much luck with it as a pure generator. Autoregressive Diffusion models (https://openreview.net/forum?id=Lm8T39vLDTE https://openreview.net/forum?id=Lm8T39vLDTE) have similar properties, but I haven't had time to sus out the subtle differences yet.
- kastnerkyle 4y ago
  Also, if you are seeking more melodies / catchy tunes to model NES-MDB (https://github.com/chrisdonahue/nesmdb https://github.com/chrisdonahue/nesmdb) might fit the bill. The modeling paper using that dataset + transformer XL - LakhNES (https://cseweb.ucsd.edu/~jmcauley/pdfs/ismir19.pdf https://cseweb.ucsd.edu/~jmcauley/pdfs/ismir19.pdf) is a pretty nice paper, with interesting samples. I have been wondering what a diffusion based take on this data might sound like, and it might be a good fit if you are looking for stuff to try.
  - zone411 4y ago
    Thanks! I wasn't familiar with a couple of these, I will definitely check them out.
lostmsu 4y ago
Am I the only one who listened to a few, and neither turned catchy at all?

38 points by zone411 4y ago

mcphage 4y ago

After listening to the same pieces over and over again on my kids' music boxes and whatnot, I've really wanted a music box that just auto-generated new 0:15 second pieces every time instead of a static loop.

deleted 4y ago

[deleted]

stolenmerch 4y ago

This is really neat. Is there a colab or Jupyter notebook we can look at?

zone411 4y ago
Not yet. The current version is not fully automated and it requires human input, so it will take a decent amount of work and documentation to make it ready for users. It also mixes C++ with Python in a few places for performance reasons when generating melodies and I don't know if that's even possible on Colab?

cuttysnark 4y ago

This is fascinating. I've always loved the idea of a .midi melody being able to be mapped to a "vocal" track and transformed accordingly—akin to Autotune The News, but with AI/ML instead of by-hand. In a sense, the lyrics could be spoken and then applied to the .midi melody. The result would sound similar to the SoundCloud vocal OP link, but with "words".

zone411 4y ago
> In a sense, the lyrics could be spoken and then applied to the .midi melody. The result would sound similar to the SoundCloud vocal OP link, but with "words". Yes, Synthesizer V Studio can do it quite well! I've added some silly lyrics to a few of these melodies and I was quite happy with the results. I'll upload them to the same YouTube channel. Here is a cover song that somebody made: https://www.youtube.com/watch?v=ettSVy4K_DY https://www.youtube.com/watch?v=ettSVy4K_DY.

armchairhacker 4y ago

I think AI in music generation will end up being a big thing like with DALL-E for images and GPT3 for text. Possibly more because it seems like people have less intuition for creating music than they do for images and words.

electric_muse 4y ago

Nice work. Somehow this reminds me of Harvest Moon 64 music. I'm not sure I felt these were necessarily catchy (but I think that comes from repetition), but they were incredibly pleasant and interesting. Also, obligatory re: earworms: https://www.youtube.com/watch?v=iDjX4-LKqCA https://www.youtube.com/watch?v=iDjX4-LKqCA

p1esk 4y ago

Are you familiar with aiva.ai? They also use midi format. Though I thought after Jukebox everyone would switch to raw audio.

chaosprint 4y ago
https://github.com/chaosprint/RaveForce https://github.com/chaosprint/RaveForce How about both symbolic and subsymbolic?
zone411 4y ago
Yes, I am. It does many things well but last time I used it (about a year ago), I couldn't get it to do what I focused on with this project: create a catchy melodic hook that could be a base of a hit song or an earworm commercial jingle. I believe this requires an element of surprise in a melody - it's much easier to just create something pleasant-sounding, without anything fancy related to neural nets.

kastnerkyle 4y ago

I've had pretty good luck recently with a mix of SUNDAE (https://arxiv.org/abs/2112.06749 https://arxiv.org/abs/2112.06749) and coconet (https://arxiv.org/abs/1903.07227 https://arxiv.org/abs/1903.07227) and/or Music Transformer based internal models recently for modeling very small datasets of polyphonic "midified" music. Research paper hopefully soon to come... Not sure what your pipeline looks like, but those papers might be worth putting on your radar. And as you mention, symbolic music datasets are both surprisingly small, surprisingly low quality, and generally a huge pain to work with. Cool stuff - I like the sax! For anyone unfamiliar with diffusion models (and coconet / OrderlessNADE), one of the really nice properties of them as opposed to "standard" autoregressive (GPT / RNN) style models, is that you should be able to specify any part, and fill in any other part - rather than being forced to specify the "past" and predict only the "future". The coconet "doodle" is a good example of this interface at work (https://www.google.com/doodles/celebrating-johann-sebastian-bach https://www.google.com/doodles/celebrating-johann-sebastian-...) XLNet had some of this promise too (https://arxiv.org/abs/1906.08237 https://arxiv.org/abs/1906.08237) but I never had much luck with it as a pure generator. Autoregressive Diffusion models (https://openreview.net/forum?id=Lm8T39vLDTE https://openreview.net/forum?id=Lm8T39vLDTE) have similar properties, but I haven't had time to sus out the subtle differences yet.

kastnerkyle 4y ago
Also, if you are seeking more melodies / catchy tunes to model NES-MDB (https://github.com/chrisdonahue/nesmdb https://github.com/chrisdonahue/nesmdb) might fit the bill. The modeling paper using that dataset + transformer XL - LakhNES (https://cseweb.ucsd.edu/~jmcauley/pdfs/ismir19.pdf https://cseweb.ucsd.edu/~jmcauley/pdfs/ismir19.pdf) is a pretty nice paper, with interesting samples. I have been wondering what a diffusion based take on this data might sound like, and it might be a good fit if you are looking for stuff to try.
- zone411 4y ago
  Thanks! I wasn't familiar with a couple of these, I will definitely check them out.

lostmsu 4y ago

Am I the only one who listened to a few, and neither turned catchy at all?