6 ms·

MusicLM: Generating Music from Text

220 points by _4bjt 4y ago

andreyk 4y ago
Link to paper: https://arxiv.org/abs/2301.11325 https://arxiv.org/abs/2301.11325 (the one on the site seems broken) Even with the knowledge that something like this is inevitabely coming, it's pretty crazy to see the examples. The fact it can create fitting music from painting descriptions (including ones such as or Guernica) is unexpected. And it seems to generate almost naturalistic vocals with reasonable-sounding lyrics, with no embedded structure for lyrics or anything. The paper itself is pretty cool. While GPT-3 was just "GPT-2, but bigger", this seems to have required some clever combination of models. Wonder if similar ideas will be needed for longer-form text or video generation.
- deleted 4y ago
  [deleted]
robobro 4y ago
I can't seem to load any of these WAV files; any mirrors?
- thekyle 4y ago
  I found that they worked for me in Chrome, but not in Safari
vlaaad 4y ago
Ohh I want a product where I can play and experiment with it!
- bsaul 4y ago
  yeah, same here. I'm also curious to see what the average quality is like, compared to those hand-picked ones.
nprateem 4y ago
Wow that is awesome. Hopefully the first feature they'll add will be to spit out midi rather than wavs (or ML that can decompose wavs to midi)
- notahacker 4y ago
  I'm surprised that ML-based MIDI generation hasn't been done quite a bit earlier and pretty well tbh. Sound is incredibly complex, but sequences of chords and notes aligned with music theory and genre conventions have well defined and easily imitated patterns. I guess part of the reason MIDI generation hasn't been a major research focus is that toy scripts get you a lot of the way. These are pretty good, even the vocals are OK, although I wonder how much if parts of the prompt like "it may be used during a festival during two songs for a buildup" are actually adding much to the mix and the music's association with the painting descriptions is as loose as I'd expect.
  - zozbot234 4y ago
    The old Biaxial-RNN by Daniel D. Johnson generates very good output for MIDI music, albeit limited to a single keyboard-like instrument. It's available at https://github.com/danieldjohnson/biaxial-rnn-music-composition https://github.com/danieldjohnson/biaxial-rnn-music-composit... and AIUI there's a GitHub fork that forward-ports it to up-to-date versions of Python (3.x series) and Theano. Transformer models are quite a bit more computationally intensive than the LSTM this used, and GPT adds attention mechanisms; but the basic approach is loosely comparable and the LSTM model can be easily trained on a single machine.
    - gavinray 4y ago
      Know of anything like this that can do EDM/House music?
- pclmulqdq 4y ago
  That's a very hard thing for them to add given the way the model is trained. It looks more like a 1-D version of an image generator than past attempts at music AI, which generated MIDI.
survirtual 4y ago
This is unbelievable. I would have never expected AI to take the jobs of creatives first, but here we are. Artists and musicians can create much better work than AI, but the AI creates consistent works usable for people who aren’t artists, and need assets for their projects — whether video games, prototyping, storyboarding, etc. It is going to be interesting to see artists revert to being a no-revenue passion again instead of being well compensated. That “well compensated” did not last long for them and I feel for the struggle. At the same time, a part of me feels art has no place being motivated by money anyway. Perhaps this change will restore the balance. Artists will need to get real jobs again like the rest of us and fund their art as a side project. I hope an open model for music generation will be soon to follow, it will really be a big deal.
- azaras 4y ago
  > It is going to be interesting to see artists revert to being a no-revenue passion again instead of being well compensated. Art should be a communication way, a hobby... All the people should have time to do art, do love or whatever they want... Also, all the people, who are able, should do productivity work, to maintain the physical bodies.
- notahacker 4y ago
  > It is going to be interesting to see artists revert to being a no-revenue passion again instead of being well compensated. Huh? Music already is a near-zero revenue passion for the vast majority of artists, has been since mainstream access to recorded music if not even earlier, and the well compensated minority's superstar status, royalties for hit songs and sellout tours aren't really threatened by DIY tools that generate nice-sounding, genre-appropriate noise a bit more efficiently than a DAW and sample pack.
- Juliate 4y ago
  > the AI creates consistent works usable for people who aren’t artists, and need assets for their projects Right but those will become more and more available, dull, and not original (because feeding from the same « repository »). So to differentiate and attract/inform/delight/surprise/tell better, they might still have to go to specialized people. Or find a way for those models to, like, hallucinate in different ways. Virtually anyone today can make their own jewels or even setup a shop on Etsy. At low cost, low price. Still, there’s a huge market for high quality and original/custom-made jewels. Especially when you want something with a story within it. And kind of a human relationship to the provider too.
bulbosaur123 4y ago
I'm really excited to hear this has been released so soon after Riffusion. Perhaps both ycombinator threads on MusicLM can be merged together. If not, I will just post my comments here. I've always seen music as the "final frontier". With images and text you can afford tiny errors and noise that isn't that noticeable by the human eye, but with music everything has to be impeccable and if a note is slightly off, you can instantly tell. Machine learning models that will be able to create outstanding music, imo, will mark the end-game of creative AI endeavors. Anyone want to share their experiments with MusicLM, feel free to join community-fan discord and subreddit: https://reddit.com/r/MusicLM/ https://reddit.com/r/MusicLM/ https://discord.gg/pjVcsyfCJR https://discord.gg/pjVcsyfCJR BTW, I really hope this gets integrated in software like Ableton similarly how image generators get slowly integrated into Adobe Creative Suite.
rany_ 4y ago
It's just disappointing that Google has a ton of cool AI projects that have nice announcement pages and seem promising but are never available to the general public.
college_physics 4y ago
Art in all its forms had always serious problems getting funded but things only seem to get worse at each iteration. In older times the problem was that most people had other worries, so artist numbers were small and they seeked patronage from the elites that could afford it. Later we had mass commercialization which "democratized" art but artists got squeezed by the intermediaries / distributors that enabled the mass market. As digital mass reproduction became a thing, this ushered zero marginal cost dynamics which is essentially still an unresolved stalemate. The "AI" generation of all sorts of combinatoric works without attribution or payment comes to compound the challenges. You now have arbitrary amounts of derivative art that can be reproduced at zero cost. Never a dull moment in homo sapiens land.
- BeFlatXIII 4y ago
  Perhaps the lesson is that art ought to be done because it is inherently fulfilling and not a as a profession.
  - necrotic_comp 4y ago
    One could say the same thing about programming. If I'm playing or writing music for people, and they are enjoying it, I want to get paid.
    - nivenkos 4y ago
      > If I'm playing or writing music for people... The whole point is that you won't be. Like a lot of demand for small-time live music died with the record player.
  - anigbrowl 4y ago
    Great, go tell landlords, grocery stores etc. It takes non-trivial investments of time to develop artistic capability, often with the aid of tuition or materials/equipment which also cost money. This is common knowledge and I frankly don't understand the point of ignoring it.
- btown 4y ago
Aissen 4y ago
A bit sad to see that the audio in the (great!) dataset consists of links to YouTube videos with timecodes, knowing the war on youtube-dl that happened just last year (long live yt-dlp !). Not really future-proof either.
gigel82 4y ago
This reminded me of DirectMusic; I remember the demos back in early 2000s (I think it was Age of Empires) where the music was dynamically generated based on what you were doing in the game (smooth transitions and everything). I wonder why that never went anywhere.
deleted 4y ago
[deleted]
legendofbrando 4y ago
I appreciate Google’s research, but it does feel like they’re behind the bar on demos/products consistently. A landing page and a research paper aren’t the best way to demonstrate software.
rhacker 4y ago
Everyone better get into construction soon. AI will replace drafters/architects/engineers but the walls going up is going to be a huge payout as Google ML engineers needing a 3rd or 4th house for their second family.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
72deluxe 4y ago
I think we need not worry. Under "Musician Experience Level" with "professional guitar player", it still sounds like a cat being strangled.

220 points by _4bjt 4y ago

Link to paper: https://arxiv.org/abs/2301.11325 https://arxiv.org/abs/2301.11325 (the one on the site seems broken) Even with the knowledge that something like this is inevitabely coming, it's pretty crazy to see the examples. The fact it can create fitting music from painting descriptions (including ones such as or Guernica) is unexpected. And it seems to generate almost naturalistic vocals with reasonable-sounding lyrics, with no embedded structure for lyrics or anything. The paper itself is pretty cool. While GPT-3 was just "GPT-2, but bigger", this seems to have required some clever combination of models. Wonder if similar ideas will be needed for longer-form text or video generation.

deleted 4y ago
[deleted]

robobro 4y ago

I can't seem to load any of these WAV files; any mirrors?

thekyle 4y ago
I found that they worked for me in Chrome, but not in Safari

vlaaad 4y ago

Ohh I want a product where I can play and experiment with it!

bsaul 4y ago
yeah, same here. I'm also curious to see what the average quality is like, compared to those hand-picked ones.

nprateem 4y ago

Wow that is awesome. Hopefully the first feature they'll add will be to spit out midi rather than wavs (or ML that can decompose wavs to midi)

notahacker 4y ago
I'm surprised that ML-based MIDI generation hasn't been done quite a bit earlier and pretty well tbh. Sound is incredibly complex, but sequences of chords and notes aligned with music theory and genre conventions have well defined and easily imitated patterns. I guess part of the reason MIDI generation hasn't been a major research focus is that toy scripts get you a lot of the way. These are pretty good, even the vocals are OK, although I wonder how much if parts of the prompt like "it may be used during a festival during two songs for a buildup" are actually adding much to the mix and the music's association with the painting descriptions is as loose as I'd expect.
- zozbot234 4y ago
  The old Biaxial-RNN by Daniel D. Johnson generates very good output for MIDI music, albeit limited to a single keyboard-like instrument. It's available at https://github.com/danieldjohnson/biaxial-rnn-music-composition https://github.com/danieldjohnson/biaxial-rnn-music-composit... and AIUI there's a GitHub fork that forward-ports it to up-to-date versions of Python (3.x series) and Theano. Transformer models are quite a bit more computationally intensive than the LSTM this used, and GPT adds attention mechanisms; but the basic approach is loosely comparable and the LSTM model can be easily trained on a single machine.
  - gavinray 4y ago
    Know of anything like this that can do EDM/House music?
pclmulqdq 4y ago
That's a very hard thing for them to add given the way the model is trained. It looks more like a 1-D version of an image generator than past attempts at music AI, which generated MIDI.

survirtual 4y ago

This is unbelievable. I would have never expected AI to take the jobs of creatives first, but here we are. Artists and musicians can create much better work than AI, but the AI creates consistent works usable for people who aren’t artists, and need assets for their projects — whether video games, prototyping, storyboarding, etc. It is going to be interesting to see artists revert to being a no-revenue passion again instead of being well compensated. That “well compensated” did not last long for them and I feel for the struggle. At the same time, a part of me feels art has no place being motivated by money anyway. Perhaps this change will restore the balance. Artists will need to get real jobs again like the rest of us and fund their art as a side project. I hope an open model for music generation will be soon to follow, it will really be a big deal.

azaras 4y ago
> It is going to be interesting to see artists revert to being a no-revenue passion again instead of being well compensated. Art should be a communication way, a hobby... All the people should have time to do art, do love or whatever they want... Also, all the people, who are able, should do productivity work, to maintain the physical bodies.
notahacker 4y ago
> It is going to be interesting to see artists revert to being a no-revenue passion again instead of being well compensated. Huh? Music already is a near-zero revenue passion for the vast majority of artists, has been since mainstream access to recorded music if not even earlier, and the well compensated minority's superstar status, royalties for hit songs and sellout tours aren't really threatened by DIY tools that generate nice-sounding, genre-appropriate noise a bit more efficiently than a DAW and sample pack.
Juliate 4y ago
> the AI creates consistent works usable for people who aren’t artists, and need assets for their projects Right but those will become more and more available, dull, and not original (because feeding from the same « repository »). So to differentiate and attract/inform/delight/surprise/tell better, they might still have to go to specialized people. Or find a way for those models to, like, hallucinate in different ways. Virtually anyone today can make their own jewels or even setup a shop on Etsy. At low cost, low price. Still, there’s a huge market for high quality and original/custom-made jewels. Especially when you want something with a story within it. And kind of a human relationship to the provider too.

bulbosaur123 4y ago

I'm really excited to hear this has been released so soon after Riffusion. Perhaps both ycombinator threads on MusicLM can be merged together. If not, I will just post my comments here. I've always seen music as the "final frontier". With images and text you can afford tiny errors and noise that isn't that noticeable by the human eye, but with music everything has to be impeccable and if a note is slightly off, you can instantly tell. Machine learning models that will be able to create outstanding music, imo, will mark the end-game of creative AI endeavors. Anyone want to share their experiments with MusicLM, feel free to join community-fan discord and subreddit: https://reddit.com/r/MusicLM/ https://reddit.com/r/MusicLM/ https://discord.gg/pjVcsyfCJR https://discord.gg/pjVcsyfCJR BTW, I really hope this gets integrated in software like Ableton similarly how image generators get slowly integrated into Adobe Creative Suite.

rany_ 4y ago

It's just disappointing that Google has a ton of cool AI projects that have nice announcement pages and seem promising but are never available to the general public.

college_physics 4y ago

Art in all its forms had always serious problems getting funded but things only seem to get worse at each iteration. In older times the problem was that most people had other worries, so artist numbers were small and they seeked patronage from the elites that could afford it. Later we had mass commercialization which "democratized" art but artists got squeezed by the intermediaries / distributors that enabled the mass market. As digital mass reproduction became a thing, this ushered zero marginal cost dynamics which is essentially still an unresolved stalemate. The "AI" generation of all sorts of combinatoric works without attribution or payment comes to compound the challenges. You now have arbitrary amounts of derivative art that can be reproduced at zero cost. Never a dull moment in homo sapiens land.

BeFlatXIII 4y ago
Perhaps the lesson is that art ought to be done because it is inherently fulfilling and not a as a profession.
- necrotic_comp 4y ago
  One could say the same thing about programming. If I'm playing or writing music for people, and they are enjoying it, I want to get paid.
  - nivenkos 4y ago
    > If I'm playing or writing music for people... The whole point is that you won't be. Like a lot of demand for small-time live music died with the record player.
- anigbrowl 4y ago
  Great, go tell landlords, grocery stores etc. It takes non-trivial investments of time to develop artistic capability, often with the aid of tuition or materials/equipment which also cost money. This is common knowledge and I frankly don't understand the point of ignoring it.
btown 4y ago

Aissen 4y ago

A bit sad to see that the audio in the (great!) dataset consists of links to YouTube videos with timecodes, knowing the war on youtube-dl that happened just last year (long live yt-dlp !). Not really future-proof either.

gigel82 4y ago

This reminded me of DirectMusic; I remember the demos back in early 2000s (I think it was Age of Empires) where the music was dynamically generated based on what you were doing in the game (smooth transitions and everything). I wonder why that never went anywhere.

deleted 4y ago

[deleted]

legendofbrando 4y ago

I appreciate Google’s research, but it does feel like they’re behind the bar on demos/products consistently. A landing page and a research paper aren’t the best way to demonstrate software.

rhacker 4y ago

Everyone better get into construction soon. AI will replace drafters/architects/engineers but the walls going up is going to be a huge payout as Google ML engineers needing a 3rd or 4th house for their second family.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

72deluxe 4y ago

I think we need not worry. Under "Musician Experience Level" with "professional guitar player", it still sounds like a cat being strangled.