8 ms·

Generating Music Tracks with Unified Representation and Diffusion Framework

96 points by laurieg 3y ago

satellite2 3y ago
So it's basically a midi predicting LLM? I don't understand how it solves the issue of representing music, i.e. actual sound waves and not simply the notes that make it up?
- theage 3y ago
  It's the best note mixing LLM I've seen. Someone just needs to train a sound diffusion model on the midi notes to orchestral mp3s. Give it a few weeks. "Unified" seems to mean all the notes in one place which you can edit/build on/mix so that you can put another layer on top, human or ai performs it with words & flare.
  - cjohnson318 3y ago
    Uh, yeah. I've been looking at programmatically writing guide tone lines and motifs in jazz for my personal piano practice, and this... looks very sophisticated.
downWidOutaFite 3y ago
I can't help but mourn a little for humanity everytime I see a demo like this one that leave little doubt that the machines will be taking away another one of our creative passions.
- arvinsim 3y ago
  If you talk about monetizing these creative pursuits, then yes, there is a reason to be worried. But if you are talking about pursuing these creative passions as an end by itself, it's much less relevant.
  - ablation 3y ago
    Quite so. Much in the same way ChatGPT or other LLMs won’t stop me writing for pleasure, entertainment or otherwise, no model will likely ever make me not want to pick up my guitar or fire up my DAW.
    - kfrzcode 3y ago
      GPT-4 in particular has, for me, rekindled a fire for writing, as I find it a compelling conversationalist quite adept to my often hyperactive, overly loquacious writing style whereas most meatspace agents find it laborious, self-aggrandizing, indulgent, and unnecessarily complex, like post-modernist novels or rationalist fanfiction. I've been engaging with GPT4 to build out an entire world of alternative hard science fiction history and it's going to be a best seller, I assure you, with zero hesitancy and full confidence. It's my idea, there's simply something so enthralling about having a "super editor" when primed with very specific prompts and understanding the boundaries of what it can't do... I'm literally 15k words into a first draft. Vomiting my ideas into words, processed through GPT to garner it's insights and suggestions, but never asking GPT to compose first drafts, has been very fun.
    - Juliate 3y ago
      But who will build and maintain a DAW (or the equivalent that enables private practice) if there is not enough people able/willing to support it? Those AI tools will be at the center of how we create and access music in the next decade. It’s a bit like cars replacing horses. There will still be horses. But transportation, city/traffic rules will adjust to cars, not horses anymore.
      - zirgs 3y ago
        Enthusiasts with AI-assisted development tools. Just like it's done with any other "historic" tech nowadays. Game console emulators, for example. Also those tools will most likely be open source.
    - widdershins 3y ago
      Yes, that's true for our generation, since we grew up with authors and musicians as role models we admired, and we loved their art and wanted to emulate it. But what will happen in a generation or two, when the world is flooded with AI generated art? Will the children of that era feel any desire to write or compose for pleasure? I'm not sure; I hope they will though.
- nxpnsv 3y ago
  There millions of musicians way more accomplished than me. But in no way do they stop me from enjoying noodling around with sound. Meanwhile, a lot of our most famous artists employ massive teams to helping them make tracks. Yet, still there are passionate creatives doing amazing stuff. I don't think creativity is over quite yet.
  - feoren 3y ago
    Yes, all this AI generated content is only more, and only adds to the total body of creative works that humans can enjoy (if you're rich). Humans are still free to pursue their own creative passions, and can get the same fulfillment from them as before (if you're rich). We can all look forward to a future where even more of our needs are met even more cheaply and easily (if you're rich). Sure, of all of the pathways previously available for humans to make a decent living and actually put a roof over their heads and food on their tables, most of them will no longer an option in a decade or two -- it was already extremely hard to make a living as a creative, and now it will be completely impossible. Nor will anyone be able to make a living driving a truck, doing landscape design, writing ad copy, writing basic code, working in a factory, optimizing a factory, designing a factory, or building a factory. Only owning things will make you money; everyone else will be competing tooth-and-nail for the few remaining ways to serve you that still require a human. Obviously those poorsies won't be able to pursue their creative passions anymore, since they're working 80 hours per week in the vain hope that their children will be able to escape the cycle of poverty. But fuck those guys: you're rich! You can play guitar all fuckin' day if you want. You are rich, right? Right?
    - nxpnsv 3y ago
      You had enough time to write this elaborate reply to my post, you must be loaded...
      - feoren 3y ago
        > in a decade or two Argument: AI has a possibility of destroying everyone's jobs so they can no longer make a living and the world is extremely uncaring toward people with no income and so they will die penniless on the street and those that don't will be heavily exploited by the rich. Counter-argument: But they can still sit around playing music if they want! No, not if they're indentured servants. I'm not saying AI will destroy most jobs, but I am saying that if it does, the fact that we can still pursue our own creative desires independent of what the AI is doing is missing the point. We've created a world where an explosion of AI capabilities would allow the rich to no longer need the poor, and they will 100% exploit that. It's not AI's fault, it's the worship of the rich and the false equivalence that wealth == human value. If someone is lamenting that there's now no chance they can make money by being a musician, telling them "whatever, just do what you love for free instead, and wait tables for 80 hours per week for an absolute pittance to make money" is not comforting.
- jjallen 3y ago
  And give us myriad new forms of media. Sort of like the advent of cars eliminating the lives of horses opening up journeys that we couldn’t even imagine before cars
- josephg 3y ago
  Judging from the output, we have at least a couple generations of these algorithms before humans have anything to worry about. The music it generates is .. musical. But it sounds bland and soulless. I struggled to get through any of the examples. I can imagine it being used to "autofill" parts of an arrangement when musicians are feeling uninspired, but I doubt it will be writing new pop songs for at least a few more years. (Though a few years is an utter drop in the bucket for humanity.)
  - zirgs 3y ago
    Bland and soulless - so just like Spotify top 50? When I listen to it - it all sounds so samey and generic that it might as well be procedurally generated.
    - josephg 3y ago
      I’ve had spotify for years and I don’t think I’ve ever listened to their top 50 playlist. Why do you listen to music you don’t like? The average song I listen to on Spotify is wildly better than what this ai is producing.
      - zirgs 3y ago
        Most people seem to like middle of the road pop that consists of the same 4 chords. I would not be surprised if AI makes stuff like that in the future. It's already formulaic, safe, marketable and non-controversial...
        tjr 3y ago
        Much commercially viable music is indeed pretty formulaic... so much so, that I wonder why we would even need something as sophisticated as "AI" to autogenerate it.
        josephg 3y ago
        This is a total hot take, but my explanation is that music - even pop music - needs to do something new or slightly different from other songs you've heard before to make it worth listening to. And having "swagger" / something unique / an interesting point of view isn't something procedural generation (or diffusion based generation in general) seems to be that good at inventing. Look at minecraft. Every map is randomly generated, but they all look kind of the same. Its really obvious in stable diffusion - people drawn by stable diffusion always have a slightly soulless look, where the quirks and charisma of the subject have been sanded down to the point of being unmemorable. I think the best music (just like the most productive artists) will soon be humans working together with AIs. Chess went through a period like this, where the best chess games were played by humans collaborating with AIs. But I think it'll be a few years yet before you get better art without humans being involved at all.
- williamcotton 3y ago
  Now you know how every musician within a 5 mile radius of you feels when Taylor Swift rolls into town with 40 trucks full of blinking lights and speaker arrays!
deleted 3y ago
[deleted]
rbflx1 3y ago
What method are they using to check to what extent generations are reproducing training examples? Because at least the "Bach Style Generation" on the website seems to just be a sped-up version of the "Befiehl du deine Wege" choral from St. Matthews Passion.
tudorw 3y ago
that a next level Rik roll right there ;)
bandika 3y ago
Musically, those examples are really bland, and the rendering is of poor quality. Besides, I'm not sure what's the point of this. I can see how ChatGPT can be useful for devs/writers/researchers etc. I can't imagine how this could be useful for a musician.
- AlecSchueler 3y ago
  To be fair some people have already listed examples of how they could use the technology elsewhere in the thread, and it will only improve from this point.
- laratied 3y ago
  I can't wait for better music AI but these examples are absolutely pathetic. It is shameful to mention Bach's name in that one example.
  - ad8e 3y ago
    Contrast this parent comment's opinion, that it is "shameful to mention Bach's name" with the Bach example, with this comment https://news.ycombinator.com/item?id=36043384 https://news.ycombinator.com/item?id=36043384, pointing out that the Bach example is plagiarized from Bach: https://www.youtube.com/watch?v=Fpqd1gCzLN4&t=178s https://www.youtube.com/watch?v=Fpqd1gCzLN4&t=178s
frankling_ 3y ago
That "Bach style generation" is on point. In fact, it's "O Haupt voll Blut und Wunden": https://www.youtube.com/watch?v=Fpqd1gCzLN4 https://www.youtube.com/watch?v=Fpqd1gCzLN4
rerdavies 3y ago
Why would anyone play the output on a awful FM synth dating to the early 80s? Somebody buy those guys a CASIO! Please!
midenginedcoupe 3y ago
Why on earth are most of these transposed? The written parts are a semitone out!? The first video is played at the same pitch as what's written, but the next two are a semitone out and Hey Jude is a whole tone away. Bizarre.

96 points by laurieg 3y ago

So it's basically a midi predicting LLM? I don't understand how it solves the issue of representing music, i.e. actual sound waves and not simply the notes that make it up?

theage 3y ago
It's the best note mixing LLM I've seen. Someone just needs to train a sound diffusion model on the midi notes to orchestral mp3s. Give it a few weeks. "Unified" seems to mean all the notes in one place which you can edit/build on/mix so that you can put another layer on top, human or ai performs it with words & flare.
- cjohnson318 3y ago
  Uh, yeah. I've been looking at programmatically writing guide tone lines and motifs in jazz for my personal piano practice, and this... looks very sophisticated.

downWidOutaFite 3y ago

I can't help but mourn a little for humanity everytime I see a demo like this one that leave little doubt that the machines will be taking away another one of our creative passions.

arvinsim 3y ago
If you talk about monetizing these creative pursuits, then yes, there is a reason to be worried. But if you are talking about pursuing these creative passions as an end by itself, it's much less relevant.
- ablation 3y ago
  Quite so. Much in the same way ChatGPT or other LLMs won’t stop me writing for pleasure, entertainment or otherwise, no model will likely ever make me not want to pick up my guitar or fire up my DAW.
  - kfrzcode 3y ago
    GPT-4 in particular has, for me, rekindled a fire for writing, as I find it a compelling conversationalist quite adept to my often hyperactive, overly loquacious writing style whereas most meatspace agents find it laborious, self-aggrandizing, indulgent, and unnecessarily complex, like post-modernist novels or rationalist fanfiction. I've been engaging with GPT4 to build out an entire world of alternative hard science fiction history and it's going to be a best seller, I assure you, with zero hesitancy and full confidence. It's my idea, there's simply something so enthralling about having a "super editor" when primed with very specific prompts and understanding the boundaries of what it can't do... I'm literally 15k words into a first draft. Vomiting my ideas into words, processed through GPT to garner it's insights and suggestions, but never asking GPT to compose first drafts, has been very fun.
  - Juliate 3y ago
    But who will build and maintain a DAW (or the equivalent that enables private practice) if there is not enough people able/willing to support it? Those AI tools will be at the center of how we create and access music in the next decade. It’s a bit like cars replacing horses. There will still be horses. But transportation, city/traffic rules will adjust to cars, not horses anymore.
    - zirgs 3y ago
      Enthusiasts with AI-assisted development tools. Just like it's done with any other "historic" tech nowadays. Game console emulators, for example. Also those tools will most likely be open source.
  - widdershins 3y ago
    Yes, that's true for our generation, since we grew up with authors and musicians as role models we admired, and we loved their art and wanted to emulate it. But what will happen in a generation or two, when the world is flooded with AI generated art? Will the children of that era feel any desire to write or compose for pleasure? I'm not sure; I hope they will though.
nxpnsv 3y ago
There millions of musicians way more accomplished than me. But in no way do they stop me from enjoying noodling around with sound. Meanwhile, a lot of our most famous artists employ massive teams to helping them make tracks. Yet, still there are passionate creatives doing amazing stuff. I don't think creativity is over quite yet.
- feoren 3y ago
  Yes, all this AI generated content is only more, and only adds to the total body of creative works that humans can enjoy (if you're rich). Humans are still free to pursue their own creative passions, and can get the same fulfillment from them as before (if you're rich). We can all look forward to a future where even more of our needs are met even more cheaply and easily (if you're rich). Sure, of all of the pathways previously available for humans to make a decent living and actually put a roof over their heads and food on their tables, most of them will no longer an option in a decade or two -- it was already extremely hard to make a living as a creative, and now it will be completely impossible. Nor will anyone be able to make a living driving a truck, doing landscape design, writing ad copy, writing basic code, working in a factory, optimizing a factory, designing a factory, or building a factory. Only owning things will make you money; everyone else will be competing tooth-and-nail for the few remaining ways to serve you that still require a human. Obviously those poorsies won't be able to pursue their creative passions anymore, since they're working 80 hours per week in the vain hope that their children will be able to escape the cycle of poverty. But fuck those guys: you're rich! You can play guitar all fuckin' day if you want. You are rich, right? Right?
  - nxpnsv 3y ago
    You had enough time to write this elaborate reply to my post, you must be loaded...
    - feoren 3y ago
      > in a decade or two Argument: AI has a possibility of destroying everyone's jobs so they can no longer make a living and the world is extremely uncaring toward people with no income and so they will die penniless on the street and those that don't will be heavily exploited by the rich. Counter-argument: But they can still sit around playing music if they want! No, not if they're indentured servants. I'm not saying AI will destroy most jobs, but I am saying that if it does, the fact that we can still pursue our own creative desires independent of what the AI is doing is missing the point. We've created a world where an explosion of AI capabilities would allow the rich to no longer need the poor, and they will 100% exploit that. It's not AI's fault, it's the worship of the rich and the false equivalence that wealth == human value. If someone is lamenting that there's now no chance they can make money by being a musician, telling them "whatever, just do what you love for free instead, and wait tables for 80 hours per week for an absolute pittance to make money" is not comforting.
jjallen 3y ago
And give us myriad new forms of media. Sort of like the advent of cars eliminating the lives of horses opening up journeys that we couldn’t even imagine before cars
josephg 3y ago
Judging from the output, we have at least a couple generations of these algorithms before humans have anything to worry about. The music it generates is .. musical. But it sounds bland and soulless. I struggled to get through any of the examples. I can imagine it being used to "autofill" parts of an arrangement when musicians are feeling uninspired, but I doubt it will be writing new pop songs for at least a few more years. (Though a few years is an utter drop in the bucket for humanity.)
- zirgs 3y ago
  Bland and soulless - so just like Spotify top 50? When I listen to it - it all sounds so samey and generic that it might as well be procedurally generated.
  - josephg 3y ago
    I’ve had spotify for years and I don’t think I’ve ever listened to their top 50 playlist. Why do you listen to music you don’t like? The average song I listen to on Spotify is wildly better than what this ai is producing.
    - zirgs 3y ago
      Most people seem to like middle of the road pop that consists of the same 4 chords. I would not be surprised if AI makes stuff like that in the future. It's already formulaic, safe, marketable and non-controversial...
      - tjr 3y ago
        Much commercially viable music is indeed pretty formulaic... so much so, that I wonder why we would even need something as sophisticated as "AI" to autogenerate it.
        josephg 3y ago
        This is a total hot take, but my explanation is that music - even pop music - needs to do something new or slightly different from other songs you've heard before to make it worth listening to. And having "swagger" / something unique / an interesting point of view isn't something procedural generation (or diffusion based generation in general) seems to be that good at inventing. Look at minecraft. Every map is randomly generated, but they all look kind of the same. Its really obvious in stable diffusion - people drawn by stable diffusion always have a slightly soulless look, where the quirks and charisma of the subject have been sanded down to the point of being unmemorable. I think the best music (just like the most productive artists) will soon be humans working together with AIs. Chess went through a period like this, where the best chess games were played by humans collaborating with AIs. But I think it'll be a few years yet before you get better art without humans being involved at all.
williamcotton 3y ago
Now you know how every musician within a 5 mile radius of you feels when Taylor Swift rolls into town with 40 trucks full of blinking lights and speaker arrays!

deleted 3y ago

[deleted]

rbflx1 3y ago

What method are they using to check to what extent generations are reproducing training examples? Because at least the "Bach Style Generation" on the website seems to just be a sped-up version of the "Befiehl du deine Wege" choral from St. Matthews Passion.

tudorw 3y ago

that a next level Rik roll right there ;)

bandika 3y ago

Musically, those examples are really bland, and the rendering is of poor quality. Besides, I'm not sure what's the point of this. I can see how ChatGPT can be useful for devs/writers/researchers etc. I can't imagine how this could be useful for a musician.

AlecSchueler 3y ago
To be fair some people have already listed examples of how they could use the technology elsewhere in the thread, and it will only improve from this point.
laratied 3y ago
I can't wait for better music AI but these examples are absolutely pathetic. It is shameful to mention Bach's name in that one example.
- ad8e 3y ago
  Contrast this parent comment's opinion, that it is "shameful to mention Bach's name" with the Bach example, with this comment https://news.ycombinator.com/item?id=36043384 https://news.ycombinator.com/item?id=36043384, pointing out that the Bach example is plagiarized from Bach: https://www.youtube.com/watch?v=Fpqd1gCzLN4&t=178s https://www.youtube.com/watch?v=Fpqd1gCzLN4&t=178s

frankling_ 3y ago

That "Bach style generation" is on point. In fact, it's "O Haupt voll Blut und Wunden": https://www.youtube.com/watch?v=Fpqd1gCzLN4 https://www.youtube.com/watch?v=Fpqd1gCzLN4

rerdavies 3y ago

Why would anyone play the output on a awful FM synth dating to the early 80s? Somebody buy those guys a CASIO! Please!

midenginedcoupe 3y ago

Why on earth are most of these transposed? The written parts are a semitone out!? The first video is played at the same pitch as what's written, but the next two are a semitone out and Hey Jude is a whole tone away. Bizarre.