10 ms·

MusicLM: Generating music from text

291 points by georgehill 4y ago

georgehill 4y ago
Demo: https://google-research.github.io/seanet/musiclm/examples/ https://google-research.github.io/seanet/musiclm/examples/
- zan2434 4y ago
  This should replace the OP link
- winrid 4y ago
  The vocals (on second page of table) are really interesting. If you walked into a club, you might not notice they are nonsense, but it's just good enough BS to be convincing.
  - oidar 4y ago
    I just popped in to mention that part. It kinda feels like when Dalle generates text - like from a distance you recognize it, but it is utterly gibberish. It's interesting because there are certain parts of machine generated outputs we zero in on - for image generation it's typically eyes and, if it has it, text. It will be interesting to watch what evolves as tell-tale sign for computer generated things.
    - LukaszWiktor 4y ago
      Eyes or rather hands?
- modeless 4y ago
  I love that there's a whole section for accordion examples. I think the accordion rap has a bad word in it. Accordion techno works surprisingly well. They really need to bump up to 48 KHz so all the music doesn't sound like it's being played over a telephone. A factor of two in the cost shouldn't be prohibitive. So much of the audio generation stuff I've seen has fatal flaws like this baked into the dataset and/or training process that ensure the output can't sound good even in theory, it's kinda frustrating. It's also frustrating that nobody AFAIK has trained a music model on an actually large dataset. We're training large language models on a significant fraction of the text of the whole internet! Where are the audio models trained on a significant fraction of all recorded music? This one was trained (in part) on a dataset of 280k hours of music, if I read the paper correctly. I don't think that comes anywhere close to being a significant fraction of all recorded music.
  - jonas-w 4y ago
    The "problem" is copyright, the music industry has way more power than the people that were copyright infringed by Stable Diffusion et al. And probably no artist that makes a living from their music will voluntarely give his music to AI.
    - makingstuffs 4y ago
      They should be able to get a significant amount of royalty free music. Granted it will generally be of a lower quality and/or be older but it should still be more than significant.
    - 5040 4y ago
      Pirates will gladly provide that content to the AI. Look what's happening with deepfakes.
  - sterlind 4y ago
    are there upscalers in the wavenet world like there are in convnets?
  - adriand 4y ago
    This is the best music that incorporates AI that I have heard by a long shot - process is here: https://ooo.ghostbows.ooo/about/ https://ooo.ghostbows.ooo/about/ Music is on all the usual streaming platforms, the album is called Shadow Planet, band is The Cotton Modules. Also avail on their website: https://ooo.ghostbows.ooo/ https://ooo.ghostbows.ooo/
- beefman 4y ago
  Totally fails on "swing" and "relaxing jazz".
- redox99 4y ago
  MusicCaps dataset: https://www.kaggle.com/datasets/googleai/musiccaps https://www.kaggle.com/datasets/googleai/musiccaps
- deleted 4y ago
  [deleted]
- isoprophlex 4y ago
  Holy hell those are insanely good. Best generated music I've heard so far. I was born too soon to explore the stars, born too late to be a 17th century pirate... but born just in time to explore these incredible neural networks come to life.
- epistemer 4y ago
  Epic soundtrack using orchestral instruments is somewhat interesting as electronic music but to me everything else is pretty bad. There is a million Industrial techno sounds, repetitive, hypnotic rhythms IDM tracks that absolutely no one listens to anymore. Most the founders of the whole genre have moved on from lack of interest. We have been able to do really good generative music in Reaktor for the last 20 years. Much better than anything on this page but no one cares. People in general want to hear the same slight variation in music over and over as background noise. The people that are saying how great all this is will be bored with it in 2 weeks or less. Pure technological kitsch.
  - CuriouslyC 4y ago
    Let's be clear, this is a tech demo, the application is not going to be prompting infinite streams of low quality music. The application of this stuff is to enable creative tools (got a beat and some vocals, give me a bassline and backing chorus), real time computer jam partners, to provide "good enough" music for indie games/movies. These models are still in their early phases - If AI was combustion engines, we wouldn't even be to railroads yet. Despite that, they're still fun to play with and useful when used correctly.
    - tomdell 4y ago
      There are tons of relatively cheap music libraries with human-composed and human-produced music that is vastly higher quality than MusicLM's output - they are used for documentaries/commercials/low-budget projects and are loaded with metadata that make it easy for directors and editors to quickly find what they are looking for. Low-quality, weird-sounding AI-generated music with wonky rhythms, an inconsistent beat, bizarre digital artifacts and a generally unpleasant lo-fi sound that probably has to be patiently generated and listened to over and over again to find a usable result is highly unlikely to beat out already very cheap, much higher-quality and easily searchable human-written music for even the lowest budget projects.
  - chasing 4y ago
    Yeah, as far as electronic music goes, maybe people don't realize just how good (and automated) the current tools are. And have been for years.
- whoomp12342 4y ago
  dope, can it queue up weird al? I am curious how generative would work at coming up with parody
dyno12345 4y ago
I want to do the reverse: pass in some music and have it describe what I'm hearing
- brookst 4y ago
  Yes! And not just notes or chords or time signatures, but real analysis of how the piece works.
  - deleted 4y ago
    [deleted]
- MadDemon 4y ago
  https://samplab.com https://samplab.com is doing something like that.
- dimatura 4y ago
  I believe parts of this model could be adapted to accomplish this. The "mulan" network generates embeddings from music and text so that they "match" each other, i.e. audio segments and text captions that go well together would be mapped to the same embedding. At run-time, they use embeddings from the text to generate the music. So conceivably the opposite could be done, using the "mulan" embeddings from audio only to generate text, by replacing the "soundstream" decoder with a text LLM. That said, there's probably simpler methods that could generate decent results, and I'd be surprised if there isn't already some out there. It's worth noting that along with this work they released a dataset of music with captions that could also help training new models for this task.
- ilaria_m 4y ago
  MusCaps: Generating Captions for Music Audio: https://arxiv.org/abs/2104.11984 https://arxiv.org/abs/2104.11984
- 52-6F-62 4y ago
  I think you’re missing the point. Music is an experience, not a checkbox or text book needing summarized.
TedDoesntTalk 4y ago
I’m writing my second video game (old school arcade style side-shooter) and hope I can use this to generate background chip tune music!
- snowram 4y ago
  May I recommend to learn how to use a tracker software? It is very fun to play with!
  - highhedgehog 4y ago
    what's a tracker software?
    - Sardtok 4y ago
      It's software used to create music. Typically you build sections that you loop and intermingle. It's kind of like writing small sections of sheet music for multiple instruments at a time, but unlike many Midi composers, the notes aren't shown as notes. It's basically what came after manually programming all the chiptunes as part of the game's code. Instead using general parsing of a format that described the audio in terms of pitch, volume, dropoff, echo, etc. of samples (at least samples were usually used with later trackers anyway).
    - danwills 4y ago
      A 'tracker' is a usually-sample-based music making software. AFAIK they were originally developed by demoscene groups (concurrently to game studios that used the code privately whereas the demogroups would release binaries for others to use for music-making with few limitations) These 'sample mixing engines' were developed in order to be able to add music to their productions (demos and games), and it became it's own whole-world. Fully agree with sibling comments, module trackers can be huge fun! Near zero barriers-to-entry.. and there's still some great stuff being made in this format too!
    - b0afc375b5 4y ago
      https://youtu.be/roBkg-iPrbw https://youtu.be/roBkg-iPrbw I think this showed up on my recommendation. I didn't know anything about tracker music, but this video explains it chronologically and very well.
winrid 4y ago
It would be cool to have a music service trained on your music playlist and just generates music you might like. Also, is this technically a vocoloid? :)
TaupeRanger 4y ago
Yet another cherry picked attempt at music gen with a "demo" page that only contains the outputs that happen to not sound like incoherent noise. There's a reason ChatGPT and Midjourney are so popular, but no music-gen tools have even come close: you can actually create stuff with them that is useful and/or enjoyable. Good music gen is much harder, and the reasons for this (still unclear) are pretty important to the future of AI, imo.
- not-chatgpt 4y ago
  Their current demos seem worse than riffusion results I get on average. Music gen is hard because music is inherently a composed of many many different instruments, each with unique sound and function. Simply training end-to-end will almost always end up badly.
  - robitsT6 4y ago
    I'm sorry, I like riffusion as much as the next guy, but in what world is any riffusion example better than the literal first example on this demo page?
    - not-chatgpt 4y ago
      Might be personal preference, but I don't think those examples are good at all. The only thing I am impressed with is the story mode and conditioning. I typically use riffusion to generate swing/electropop music, could be I'm too biased.
  - zone411 4y ago
    > Music gen is hard because music is inherently a composed of many many different instruments, each with unique sound and function. Simply training end-to-end will almost always end up badly. Yes! That's what I figured when I started my project to create an AI assistant for melodies (http://melodies.ai/ http://melodies.ai/). I'm quite sure that splitting it up is the way to go, and that the first AI hit song will not be created as a full song but by combining instruments/vocals.
  - 4y ago
woolion 4y ago
I don't really understand why this approached is pushed for music. You can overpaint an image, but you can't do that with a song. Cutting an image to reintroduce coherence is easy too. For a song you need midi, or another symbolic representation. That was the approach of pop2piano (unfortunately it is limited to covers, not generating from scratch). And even if a song generated this is OK, listening to half an hour full of AI mistakes is really tiring. With a symbolic representation you could at least fix the mistakes if there is one good output.
- dimatura 4y ago
  I understand what you're saying, although it could be argued that at least for some types of image tasks one would prefer something like an SVG output with layers to make it easier to edit. For music, I think it's partly an academic question of "can we do it" rather than trying to maximize immediate practical usefulness. There's already quite a bit of work on symbolic music generation (mostly MIDI), a lot of it quite competent, especially in more constrained domains like NES chiptunes or classical piano, so a full text-to-audio pipeline probably seemed a more interesting research problem. And for a lot of use cases, where people might truly not care too much about tweaking the output to their liking, the generated audio might be good enough; the examples were pretty plausible to my ear, if somewhat lo-fi sounding (probably because it's operating at 24kHz, compared to the more standard 44-48kHz). In the future a more hybrid approach probably makes sense for at least some applications, where MIDI is generated along with some way of specifying the timbre for each instrument (hopefully something better than general MIDI, though even that would be fun; not sure if it's been done). I'm sure that in the near we'll see a lot more work in the DAW and plugin space to have these kind of things built-in, but in a way that they can be edited by the user.
  - westurner 4y ago
    awesome-sheet-music lists a number of sheet music archives https://github.com/ad-si/awesome-sheet-music https://github.com/ad-si/awesome-sheet-music Other libraries of (Royalty Free, Public Domain) sheet music: - https://musopen.org/ https://musopen.org/ Explainable artificial intelligence: https://en.wikipedia.org/wiki/Explainable_artificial_intelligence https://en.wikipedia.org/wiki/Explainable_artificial_intelli... FWIU, Current LLMs can't yet do explainable AI well enough to satisfy the optional Attribution clause of e.g. Creative Commons licenses? "Sufficiently Transformative" is the current general copyright burden according to precedent; Transformative use and fair use: https://en.wikipedia.org/wiki/Transformative_use https://en.wikipedia.org/wiki/Transformative_use
FeepingCreature 4y ago
Weights when though. :-(
- snowram 4y ago
  Knowing Google, they will somehow never released it by invoking a "think of the children", like every single one of their ML papers.
- ben_w 4y ago
  For this specific paper, probably never. From the final section: """We acknowledge the risk of potential misappropriation of creative content associated to the use-case. In accordance with responsible model development practices, we conducted a thorough study of memorization, adapting and extending a methodology used in the context of text-based LLMs, focusing on the semantic modeling stage. We found that only a tiny fraction of examples was memorized exactly, while for 1% of the examples we could identify an approximate match. We strongly emphasize the need for more future work in tackling these risks associated to music generation — we have no plans to release models at this point."""
  - spuz 4y ago
    Could someone translate this into layman's terms?
    - ben_w 4y ago
      It actually copied some stuff. Not much, but it did, and that's bad.
andjelam990 4y ago
Interesting concept! I wonder how the hinders around copywriting will be solved.
spyder 4y ago
Awesome, this is probably the best one to date even compared to the recent Riffusion, Jukebox and the older MIDI generating MuseNet. especially the conditioning on humming and whistling examples are cool, but to bad they use very common melodies for that so it's easier job for the model and harder for us to judge how well would it work on less common melodies.
- LunarAurora 4y ago
  > especially the conditioning on humming and whistling ... but to bad they use very common melodies 100%. Now if you add the detailed "Painting Caption Conditioning" to the mix, you can create a melody in the style of… an image, which, in a way, is kind of a controlled artificial synaesthesia [1]. [1] https://en.wikipedia.org/wiki/Synesthesia https://en.wikipedia.org/wiki/Synesthesia
kleiba 4y ago
Note that there's a second thread on the front page right now that links to the examples: https://news.ycombinator.com/item?id=34541693 https://news.ycombinator.com/item?id=34541693 Perhaps they could be merged?
akie 4y ago
I hate how good this is.
dariosalvi78 4y ago
AI music is the next thing coming, wait for copyright lawsuits to fall like bombs. I find it fascinating that MidJourney can make a 3D model of my face from a low quality image, rotate it in space, apply it on someone's else body, add coherent shadows and backgrounds, with a very credible result, and yet an AI cannot generate a decent song, which is 1-dimensional and has probably much less internal modelling to care about? One reason I can think of is because eyes are "integrators" and ears are "derivators". That is, that human ear is very sensitive so small differences, whereas vision cares more about the ensemble? I don't know, but I think that AI music will come one day. It may not be as great as human music, but it will suffice for, say, putting a music background for your startup cheap marketing ad.
- infinitifall 4y ago
  Possibly just anecdotal evidence, but it seems like the pool of good music composers is much much smaller than the pool of good visual artists. Maybe that is an indicator of how "hard" each field is.
  - notahacker 4y ago
    Think that depends heavily on what you consider to be a "composer" and what you consider to be "good" in both art and music. Writing somewhat novel music to a formula arguably has a much lower skill bar than producing good representative art (especially if you allow sequencers as composition and disallow basic digital retouching of photos as visual art), but producing something that genuinely stands out may be harder
- jtode 4y ago
  It will be as good as the music it's trained on, no better and probably no worse.
- 52-6F-62 4y ago
  > which is 1-dimensional and has probably much less internal modelling to care about There is an ocean between “I like that sound” and a final, produced piece of recorded music. Much of that ocean being ineffable. Trying to reduce it to a set of parameters around the final waveform and you’ve missed the entire point. > but it will suffice for, say, putting a music background for your startup cheap marketing ad. We need less of that—not more. It’s like a climate disaster of the soul.
deleted 4y ago
[deleted]
duckington 4y ago
I wonder if this technology will eventually revolutionize music the same way synthesizers did. Or at least lead to music and effects/filters that are simply not possible with current DAWs and plugins. Custom generation of samples from text alone seems revolutionary.
bulbosaur123 4y ago
I'm absolutely flabbergasted by rapid progression of music related machine learning models. I've always seen music as the "final frontier". With images you can have tiny errors and noise that isn't that noticeable by the human eye, but with music everything has to be impeccable and if a note is slightly off, you instantly hear it. Machine learning models that will be able to create outstanding music, imo, will mark the end-game of creative AI. Anyone want to share their experiments with MusicLM, feel free to join community-fan discord and subreddit: https://reddit.com/r/MusicLM/ https://reddit.com/r/MusicLM/ https://discord.gg/pjVcsyfCJR https://discord.gg/pjVcsyfCJR BTW, I really hope this gets integrated in software like Ableton similarly how image generators get slowly integrated into Adobe Creative Suite.
- netr0ute 4y ago
  > if a note is slightly off, you instantly hear it I don't know about this. I know professional musicians and they say that they make little mistakes all the time, but part of being a professional is being able to pretend those mistakes didn't exist because the audience doesn't know what it's supposed to sound like.
  - dwringer 4y ago
    > but part of being a professional is being able to pretend those mistakes didn't exist because the audience doesn't know what it's supposed to sound like. Very true, however it's something musicians have to specifically train for, because to a trained ear they can become painfully obvious and it's tempting for the musician to (at least) cringe in a way that the audience will detect. Musicians in the audience, though, can often tell when you flub a note, regardless of how you play it off. That's really the bar we are looking to meet.
- jameshart 4y ago
  The fact that mistakes in music are so obvious comes down the the fact that underneath music is a lot of structure and pattern. That there can be a ‘wrong’ note implies that there is redundancy in the signal and predicting the right next sound is precisely the kind of thing these sorts of generative AIs excel at.
deleted 4y ago
[deleted]
guyisra 4y ago
Yet Another ClosedAI research project with no intention of releasing.. what a joke
williamcotton 4y ago
As a musician I don’t see these tools as competing with what I do but enabling me to do so much more. It takes a lot of time and money to create a professional recording for my songs and drum machines and synths just don’t work for Americana. These tools offer the possibility of backing tracks that sound like Willie Nelson’s band from the 70s but at a fraction of the time and effort. I can’t wait until they get to the point where they’re more composable or auto-accompany given an acoustic guitar and vocal input.
eddsh1994 4y ago
Where can I get the MusicCaps dataset? I can't seem to find it
- ptimothymooney 4y ago
  https://www.kaggle.com/datasets/googleai/musiccaps https://www.kaggle.com/datasets/googleai/musiccaps
threevox 4y ago
Google: kings of releasing papers but never shipping anything
pkchv 4y ago
I love the current rate of progress in generative music ml research. Riffusion model ( https://www.riffusion.com/about https://www.riffusion.com/about ) was trending here just a month ago. Pity Google never shares the actual models, would love to hear artists go wild with it. btw, some of you might find interesting, diffusion-based / generative concept album: https://wielu.bandcamp.com/album/vectorstep-ep https://wielu.bandcamp.com/album/vectorstep-ep
gschoeni 4y ago
Any plans on releasing the actual audio data and not just a csv with links to YouTube IDs? Also maybe a silly question, but what's the legal ramifications of downloading these YouTube videos and training on them yourself? Google must have some rights, but what about people outside of Google?
frinnylee 4y ago
Anyone knows how to reach out the team?
Gertie01 4y ago
Can't wait to generate infinite music.
xp84 4y ago
They said they're not going to allow people to use it based on fear of plagiarism accusations / music industry lawsuits. Ugh, typical. I want someone to train it on public domain music. Kind of like the YouTube Audio Library but I assume that's not exactly the right license for this. But with sufficient effort someone could make a lot of recordings of public domain music for this purpose and build something that the RIAA thugs couldn't actually touch.

291 points by georgehill 4y ago

Demo: https://google-research.github.io/seanet/musiclm/examples/ https://google-research.github.io/seanet/musiclm/examples/

zan2434 4y ago
This should replace the OP link
winrid 4y ago
The vocals (on second page of table) are really interesting. If you walked into a club, you might not notice they are nonsense, but it's just good enough BS to be convincing.
- oidar 4y ago
  I just popped in to mention that part. It kinda feels like when Dalle generates text - like from a distance you recognize it, but it is utterly gibberish. It's interesting because there are certain parts of machine generated outputs we zero in on - for image generation it's typically eyes and, if it has it, text. It will be interesting to watch what evolves as tell-tale sign for computer generated things.
  - LukaszWiktor 4y ago
    Eyes or rather hands?
modeless 4y ago
I love that there's a whole section for accordion examples. I think the accordion rap has a bad word in it. Accordion techno works surprisingly well. They really need to bump up to 48 KHz so all the music doesn't sound like it's being played over a telephone. A factor of two in the cost shouldn't be prohibitive. So much of the audio generation stuff I've seen has fatal flaws like this baked into the dataset and/or training process that ensure the output can't sound good even in theory, it's kinda frustrating. It's also frustrating that nobody AFAIK has trained a music model on an actually large dataset. We're training large language models on a significant fraction of the text of the whole internet! Where are the audio models trained on a significant fraction of all recorded music? This one was trained (in part) on a dataset of 280k hours of music, if I read the paper correctly. I don't think that comes anywhere close to being a significant fraction of all recorded music.
- jonas-w 4y ago
  The "problem" is copyright, the music industry has way more power than the people that were copyright infringed by Stable Diffusion et al. And probably no artist that makes a living from their music will voluntarely give his music to AI.
  - makingstuffs 4y ago
    They should be able to get a significant amount of royalty free music. Granted it will generally be of a lower quality and/or be older but it should still be more than significant.
  - 5040 4y ago
    Pirates will gladly provide that content to the AI. Look what's happening with deepfakes.
- sterlind 4y ago
  are there upscalers in the wavenet world like there are in convnets?
- adriand 4y ago
  This is the best music that incorporates AI that I have heard by a long shot - process is here: https://ooo.ghostbows.ooo/about/ https://ooo.ghostbows.ooo/about/ Music is on all the usual streaming platforms, the album is called Shadow Planet, band is The Cotton Modules. Also avail on their website: https://ooo.ghostbows.ooo/ https://ooo.ghostbows.ooo/
beefman 4y ago
Totally fails on "swing" and "relaxing jazz".
redox99 4y ago
MusicCaps dataset: https://www.kaggle.com/datasets/googleai/musiccaps https://www.kaggle.com/datasets/googleai/musiccaps
deleted 4y ago
[deleted]
isoprophlex 4y ago
Holy hell those are insanely good. Best generated music I've heard so far. I was born too soon to explore the stars, born too late to be a 17th century pirate... but born just in time to explore these incredible neural networks come to life.
epistemer 4y ago
Epic soundtrack using orchestral instruments is somewhat interesting as electronic music but to me everything else is pretty bad. There is a million Industrial techno sounds, repetitive, hypnotic rhythms IDM tracks that absolutely no one listens to anymore. Most the founders of the whole genre have moved on from lack of interest. We have been able to do really good generative music in Reaktor for the last 20 years. Much better than anything on this page but no one cares. People in general want to hear the same slight variation in music over and over as background noise. The people that are saying how great all this is will be bored with it in 2 weeks or less. Pure technological kitsch.
- CuriouslyC 4y ago
  Let's be clear, this is a tech demo, the application is not going to be prompting infinite streams of low quality music. The application of this stuff is to enable creative tools (got a beat and some vocals, give me a bassline and backing chorus), real time computer jam partners, to provide "good enough" music for indie games/movies. These models are still in their early phases - If AI was combustion engines, we wouldn't even be to railroads yet. Despite that, they're still fun to play with and useful when used correctly.
  - tomdell 4y ago
    There are tons of relatively cheap music libraries with human-composed and human-produced music that is vastly higher quality than MusicLM's output - they are used for documentaries/commercials/low-budget projects and are loaded with metadata that make it easy for directors and editors to quickly find what they are looking for. Low-quality, weird-sounding AI-generated music with wonky rhythms, an inconsistent beat, bizarre digital artifacts and a generally unpleasant lo-fi sound that probably has to be patiently generated and listened to over and over again to find a usable result is highly unlikely to beat out already very cheap, much higher-quality and easily searchable human-written music for even the lowest budget projects.
- chasing 4y ago
  Yeah, as far as electronic music goes, maybe people don't realize just how good (and automated) the current tools are. And have been for years.
whoomp12342 4y ago
dope, can it queue up weird al? I am curious how generative would work at coming up with parody

dyno12345 4y ago

I want to do the reverse: pass in some music and have it describe what I'm hearing

brookst 4y ago
Yes! And not just notes or chords or time signatures, but real analysis of how the piece works.
- deleted 4y ago
  [deleted]
MadDemon 4y ago
https://samplab.com https://samplab.com is doing something like that.
dimatura 4y ago
I believe parts of this model could be adapted to accomplish this. The "mulan" network generates embeddings from music and text so that they "match" each other, i.e. audio segments and text captions that go well together would be mapped to the same embedding. At run-time, they use embeddings from the text to generate the music. So conceivably the opposite could be done, using the "mulan" embeddings from audio only to generate text, by replacing the "soundstream" decoder with a text LLM. That said, there's probably simpler methods that could generate decent results, and I'd be surprised if there isn't already some out there. It's worth noting that along with this work they released a dataset of music with captions that could also help training new models for this task.
ilaria_m 4y ago
MusCaps: Generating Captions for Music Audio: https://arxiv.org/abs/2104.11984 https://arxiv.org/abs/2104.11984
52-6F-62 4y ago
I think you’re missing the point. Music is an experience, not a checkbox or text book needing summarized.

TedDoesntTalk 4y ago

I’m writing my second video game (old school arcade style side-shooter) and hope I can use this to generate background chip tune music!

snowram 4y ago
May I recommend to learn how to use a tracker software? It is very fun to play with!
- highhedgehog 4y ago
  what's a tracker software?
  - Sardtok 4y ago
    It's software used to create music. Typically you build sections that you loop and intermingle. It's kind of like writing small sections of sheet music for multiple instruments at a time, but unlike many Midi composers, the notes aren't shown as notes. It's basically what came after manually programming all the chiptunes as part of the game's code. Instead using general parsing of a format that described the audio in terms of pitch, volume, dropoff, echo, etc. of samples (at least samples were usually used with later trackers anyway).
  - danwills 4y ago
    A 'tracker' is a usually-sample-based music making software. AFAIK they were originally developed by demoscene groups (concurrently to game studios that used the code privately whereas the demogroups would release binaries for others to use for music-making with few limitations) These 'sample mixing engines' were developed in order to be able to add music to their productions (demos and games), and it became it's own whole-world. Fully agree with sibling comments, module trackers can be huge fun! Near zero barriers-to-entry.. and there's still some great stuff being made in this format too!
  - b0afc375b5 4y ago
    https://youtu.be/roBkg-iPrbw https://youtu.be/roBkg-iPrbw I think this showed up on my recommendation. I didn't know anything about tracker music, but this video explains it chronologically and very well.

winrid 4y ago

It would be cool to have a music service trained on your music playlist and just generates music you might like. Also, is this technically a vocoloid? :)

TaupeRanger 4y ago

Yet another cherry picked attempt at music gen with a "demo" page that only contains the outputs that happen to not sound like incoherent noise. There's a reason ChatGPT and Midjourney are so popular, but no music-gen tools have even come close: you can actually create stuff with them that is useful and/or enjoyable. Good music gen is much harder, and the reasons for this (still unclear) are pretty important to the future of AI, imo.

not-chatgpt 4y ago
Their current demos seem worse than riffusion results I get on average. Music gen is hard because music is inherently a composed of many many different instruments, each with unique sound and function. Simply training end-to-end will almost always end up badly.
- robitsT6 4y ago
  I'm sorry, I like riffusion as much as the next guy, but in what world is any riffusion example better than the literal first example on this demo page?
  - not-chatgpt 4y ago
    Might be personal preference, but I don't think those examples are good at all. The only thing I am impressed with is the story mode and conditioning. I typically use riffusion to generate swing/electropop music, could be I'm too biased.
- zone411 4y ago
  > Music gen is hard because music is inherently a composed of many many different instruments, each with unique sound and function. Simply training end-to-end will almost always end up badly. Yes! That's what I figured when I started my project to create an AI assistant for melodies (http://melodies.ai/ http://melodies.ai/). I'm quite sure that splitting it up is the way to go, and that the first AI hit song will not be created as a full song but by combining instruments/vocals.
- 4y ago

woolion 4y ago

I don't really understand why this approached is pushed for music. You can overpaint an image, but you can't do that with a song. Cutting an image to reintroduce coherence is easy too. For a song you need midi, or another symbolic representation. That was the approach of pop2piano (unfortunately it is limited to covers, not generating from scratch). And even if a song generated this is OK, listening to half an hour full of AI mistakes is really tiring. With a symbolic representation you could at least fix the mistakes if there is one good output.

dimatura 4y ago
I understand what you're saying, although it could be argued that at least for some types of image tasks one would prefer something like an SVG output with layers to make it easier to edit. For music, I think it's partly an academic question of "can we do it" rather than trying to maximize immediate practical usefulness. There's already quite a bit of work on symbolic music generation (mostly MIDI), a lot of it quite competent, especially in more constrained domains like NES chiptunes or classical piano, so a full text-to-audio pipeline probably seemed a more interesting research problem. And for a lot of use cases, where people might truly not care too much about tweaking the output to their liking, the generated audio might be good enough; the examples were pretty plausible to my ear, if somewhat lo-fi sounding (probably because it's operating at 24kHz, compared to the more standard 44-48kHz). In the future a more hybrid approach probably makes sense for at least some applications, where MIDI is generated along with some way of specifying the timbre for each instrument (hopefully something better than general MIDI, though even that would be fun; not sure if it's been done). I'm sure that in the near we'll see a lot more work in the DAW and plugin space to have these kind of things built-in, but in a way that they can be edited by the user.
- westurner 4y ago
  awesome-sheet-music lists a number of sheet music archives https://github.com/ad-si/awesome-sheet-music https://github.com/ad-si/awesome-sheet-music Other libraries of (Royalty Free, Public Domain) sheet music: - https://musopen.org/ https://musopen.org/ Explainable artificial intelligence: https://en.wikipedia.org/wiki/Explainable_artificial_intelligence https://en.wikipedia.org/wiki/Explainable_artificial_intelli... FWIU, Current LLMs can't yet do explainable AI well enough to satisfy the optional Attribution clause of e.g. Creative Commons licenses? "Sufficiently Transformative" is the current general copyright burden according to precedent; Transformative use and fair use: https://en.wikipedia.org/wiki/Transformative_use https://en.wikipedia.org/wiki/Transformative_use

FeepingCreature 4y ago

Weights when though. :-(

snowram 4y ago
Knowing Google, they will somehow never released it by invoking a "think of the children", like every single one of their ML papers.
ben_w 4y ago
For this specific paper, probably never. From the final section: """We acknowledge the risk of potential misappropriation of creative content associated to the use-case. In accordance with responsible model development practices, we conducted a thorough study of memorization, adapting and extending a methodology used in the context of text-based LLMs, focusing on the semantic modeling stage. We found that only a tiny fraction of examples was memorized exactly, while for 1% of the examples we could identify an approximate match. We strongly emphasize the need for more future work in tackling these risks associated to music generation — we have no plans to release models at this point."""
- spuz 4y ago
  Could someone translate this into layman's terms?
  - ben_w 4y ago
    It actually copied some stuff. Not much, but it did, and that's bad.

andjelam990 4y ago

Interesting concept! I wonder how the hinders around copywriting will be solved.

spyder 4y ago

Awesome, this is probably the best one to date even compared to the recent Riffusion, Jukebox and the older MIDI generating MuseNet. especially the conditioning on humming and whistling examples are cool, but to bad they use very common melodies for that so it's easier job for the model and harder for us to judge how well would it work on less common melodies.

LunarAurora 4y ago
> especially the conditioning on humming and whistling ... but to bad they use very common melodies 100%. Now if you add the detailed "Painting Caption Conditioning" to the mix, you can create a melody in the style of… an image, which, in a way, is kind of a controlled artificial synaesthesia [1]. [1] https://en.wikipedia.org/wiki/Synesthesia https://en.wikipedia.org/wiki/Synesthesia

kleiba 4y ago

Note that there's a second thread on the front page right now that links to the examples: https://news.ycombinator.com/item?id=34541693 https://news.ycombinator.com/item?id=34541693 Perhaps they could be merged?

akie 4y ago

I hate how good this is.

dariosalvi78 4y ago

AI music is the next thing coming, wait for copyright lawsuits to fall like bombs. I find it fascinating that MidJourney can make a 3D model of my face from a low quality image, rotate it in space, apply it on someone's else body, add coherent shadows and backgrounds, with a very credible result, and yet an AI cannot generate a decent song, which is 1-dimensional and has probably much less internal modelling to care about? One reason I can think of is because eyes are "integrators" and ears are "derivators". That is, that human ear is very sensitive so small differences, whereas vision cares more about the ensemble? I don't know, but I think that AI music will come one day. It may not be as great as human music, but it will suffice for, say, putting a music background for your startup cheap marketing ad.

infinitifall 4y ago
Possibly just anecdotal evidence, but it seems like the pool of good music composers is much much smaller than the pool of good visual artists. Maybe that is an indicator of how "hard" each field is.
- notahacker 4y ago
  Think that depends heavily on what you consider to be a "composer" and what you consider to be "good" in both art and music. Writing somewhat novel music to a formula arguably has a much lower skill bar than producing good representative art (especially if you allow sequencers as composition and disallow basic digital retouching of photos as visual art), but producing something that genuinely stands out may be harder
jtode 4y ago
It will be as good as the music it's trained on, no better and probably no worse.
52-6F-62 4y ago
> which is 1-dimensional and has probably much less internal modelling to care about There is an ocean between “I like that sound” and a final, produced piece of recorded music. Much of that ocean being ineffable. Trying to reduce it to a set of parameters around the final waveform and you’ve missed the entire point. > but it will suffice for, say, putting a music background for your startup cheap marketing ad. We need less of that—not more. It’s like a climate disaster of the soul.

deleted 4y ago

[deleted]

duckington 4y ago

I wonder if this technology will eventually revolutionize music the same way synthesizers did. Or at least lead to music and effects/filters that are simply not possible with current DAWs and plugins. Custom generation of samples from text alone seems revolutionary.

bulbosaur123 4y ago

I'm absolutely flabbergasted by rapid progression of music related machine learning models. I've always seen music as the "final frontier". With images you can have tiny errors and noise that isn't that noticeable by the human eye, but with music everything has to be impeccable and if a note is slightly off, you instantly hear it. Machine learning models that will be able to create outstanding music, imo, will mark the end-game of creative AI. Anyone want to share their experiments with MusicLM, feel free to join community-fan discord and subreddit: https://reddit.com/r/MusicLM/ https://reddit.com/r/MusicLM/ https://discord.gg/pjVcsyfCJR https://discord.gg/pjVcsyfCJR BTW, I really hope this gets integrated in software like Ableton similarly how image generators get slowly integrated into Adobe Creative Suite.

netr0ute 4y ago
> if a note is slightly off, you instantly hear it I don't know about this. I know professional musicians and they say that they make little mistakes all the time, but part of being a professional is being able to pretend those mistakes didn't exist because the audience doesn't know what it's supposed to sound like.
- dwringer 4y ago
  > but part of being a professional is being able to pretend those mistakes didn't exist because the audience doesn't know what it's supposed to sound like. Very true, however it's something musicians have to specifically train for, because to a trained ear they can become painfully obvious and it's tempting for the musician to (at least) cringe in a way that the audience will detect. Musicians in the audience, though, can often tell when you flub a note, regardless of how you play it off. That's really the bar we are looking to meet.
jameshart 4y ago
The fact that mistakes in music are so obvious comes down the the fact that underneath music is a lot of structure and pattern. That there can be a ‘wrong’ note implies that there is redundancy in the signal and predicting the right next sound is precisely the kind of thing these sorts of generative AIs excel at.

deleted 4y ago

[deleted]

guyisra 4y ago

Yet Another ClosedAI research project with no intention of releasing.. what a joke

williamcotton 4y ago

As a musician I don’t see these tools as competing with what I do but enabling me to do so much more. It takes a lot of time and money to create a professional recording for my songs and drum machines and synths just don’t work for Americana. These tools offer the possibility of backing tracks that sound like Willie Nelson’s band from the 70s but at a fraction of the time and effort. I can’t wait until they get to the point where they’re more composable or auto-accompany given an acoustic guitar and vocal input.

eddsh1994 4y ago

Where can I get the MusicCaps dataset? I can't seem to find it

ptimothymooney 4y ago
https://www.kaggle.com/datasets/googleai/musiccaps https://www.kaggle.com/datasets/googleai/musiccaps

threevox 4y ago

Google: kings of releasing papers but never shipping anything

pkchv 4y ago

I love the current rate of progress in generative music ml research. Riffusion model ( https://www.riffusion.com/about https://www.riffusion.com/about ) was trending here just a month ago. Pity Google never shares the actual models, would love to hear artists go wild with it. btw, some of you might find interesting, diffusion-based / generative concept album: https://wielu.bandcamp.com/album/vectorstep-ep https://wielu.bandcamp.com/album/vectorstep-ep

gschoeni 4y ago

Any plans on releasing the actual audio data and not just a csv with links to YouTube IDs? Also maybe a silly question, but what's the legal ramifications of downloading these YouTube videos and training on them yourself? Google must have some rights, but what about people outside of Google?

frinnylee 4y ago

Anyone knows how to reach out the team?

Gertie01 4y ago

Can't wait to generate infinite music.

xp84 4y ago

They said they're not going to allow people to use it based on fear of plagiarism accusations / music industry lawsuits. Ugh, typical. I want someone to train it on public domain music. Kind of like the YouTube Audio Library but I assume that's not exactly the right license for this. But with sufficient effort someone could make a lot of recordings of public domain music for this purpose and build something that the RIAA thugs couldn't actually touch.