6 ms·

ACE-Step: A step towards music generation foundation model

109 points by wertyk 1y ago

frankfrank13 1y ago
Fun to play with!
hammock 1y ago
Is there a demo hosted somewhere?
- frankfrank13 1y ago
  - https://ace-step.github.io/ https://ace-step.github.io/ - https://huggingface.co/spaces/ACE-Step/ACE-Step https://huggingface.co/spaces/ACE-Step/ACE-Step
Rodeoclash 1y ago
As a musician, the things I want most from generative AI is: 1. Being able to have the AI fill in a track in the song, but use the whole song as input to figure out what to generate. Ideally for drums this would be a combination of individual drum hits, effects and midi so I'm able to tweak it after generation. If it used the Ableton effects and drum rack then that would be perfect. 2. Take my singing and make it both sound great and like any combination of great singers (e.g. give me a bit of Taylor Swift combined with Cat Power) I've had a play with the style transfer between singers (bullet point 2 above) but when I last tried it, it was garbage in / garbage out, and my singing is garbage. What I don't want: To just generate a whole song. Adobe does this style of assistive AI well in the photo editing space but no one seems to have brought it to audio yet.
- pelagic_sky 1y ago
  Exactly. I don’t want AI to make me a song. I want it to switch up my baseline or chords or recommend a fill instrument.
- vunderba 1y ago
  Logic Pro comes the closest to this with the addition of virtual drummers. You can assign it to follow certain rhythmic timings by connecting it to a main instrument track, and by labeling sections of your song (bridge, chorus, etc) you can regenerate until you find something you're happy with. It's a far cry from having a real drummer, but it works in a pinch.
- viraptor 1y ago
  Spark amp is now advertising with some interesting AI options. https://au.positivegrid.com/products/spark-2 https://au.positivegrid.com/products/spark-2 As far as I understand it can attempt to generate backing drums (and music?) for jamming, which sounds great for practicing.
- dstorm1 1y ago
  I think you might like https://soundry.ai https://soundry.ai they are working on a feature that is just like 1.
- nprateem 1y ago
  I think the big limiting factor is having it learn what the infinite number of parameters in different synths, etc actually make it sound like, and then being able to make the model be able to produce music by using and configuring them instead of just spaffing out crap music. If it could do that it'd give the tweakability it needs, but I think it would basically involve training a model for each instrument.
architectonic 1y ago
How do the quality and prompt adherence compare to Suno v4?
999900000999 1y ago
> RapMachine Fine-tuned on pure rap data to create an AI system specialized in rap generation Expected capabilities include AI rap battles and narrative expression through rap Rap has exceptional storytelling and expressive capabilities, offering extraordinary application potential Using a certain other music generator I got it to accidentally say ***. It said it with a Latino American accent too. In fact for whatever reason this tool couldn’t use a typical AAVE voice. Just Sage Francis / Atmosphere like dictionary raps and a few Latino American ones. A big limitation of AI sloop is it tries to not offend anyone. Art that can’t even try to offend is barely art.
- Duwensatzaj 1y ago
  > Art that can’t even try to offend is barely art. Good art is secondary to avoiding some journalist writing a hit piece about how they used your company’s AI generator to depict Hitler saying the gamer word.
  - 999900000999 1y ago
    AI generated articles about ai generated music which is generated from reading ai generated articles. Shall it continue in an unholy loop until the end of time ?
  - lwansbrough 1y ago
    To be fair, nothing AI generated is inherently “good art” if you take artistic intent into account.
    - justlikereddit 1y ago
      Mixing paint in your rectum with the intention of making good art will always rate lower than AI slop
      - nxpnsv 1y ago
        *citation needed
dheera 1y ago
The diagram is super vague. How are the lyrics encoded? What does the encoder look like inside? What is the input size, input format, output size, output format? Are the three encoder outputs added? Concatentated? When Mert and m-Hubert combine are they added? Multiplied? Subtracted? Concatenated? I really wish people could make better diagrams.
lilcrise 1y ago
VPS 4 Cores SSD Disk Space: 200 GB CPU cores: 4 RAM: 4 GB Ubuntu Server 22.04 can it run on vps server this small?
montag 1y ago
The experimental section was most interesting: https://ace-step.github.io/#Experimental https://ace-step.github.io/#Experimental Puh tss puh tss Kshhh ff kshhh ff Bmm tkk bmm tkk Drr-dr-d-d-dt
l72 1y ago
I want to play something on my keyboard (the only instrument I am slightly ok at) and then be able to tell it to play it with a saxophone and describe exactly how I want it played. I don’t need an AI to create a song for me, I need 100 session musicians at my disposal to create the song I want. I am very excited about having that type of ai.
- spyder 1y ago
  Here is a cool demonstration to do voice-to-instrument or instrument-to-another instrument (The inconvenient thing is that for a new kind of output sound you have to train a model for around 1 hour for good quality, but after that you can use it with different inputs quickly): https://youtu.be/lI1LCfTx2lI?t=525 https://youtu.be/lI1LCfTx2lI?t=525 There is also Kits.ai https://www.kits.ai/tools/ai-instruments https://www.kits.ai/tools/ai-instruments
- arafalov 1y ago
  You sound like you are just talking about virtual instruments, which are getting better and better. Something like this maybe, doing the virtual instruments (VST/VSTI) review for saxophone: https://news.ycombinator.com/item?id=43909398 https://news.ycombinator.com/item?id=43909398
peterzhang2000 1y ago
[dead]
Alex_001 1y ago
Really interesting — we're seeing more efforts now to bring the "foundation model" approach to creative domains like music, but I wonder how well these models can internalize musical structure over long time scales. Has anyone here compared ACE-Step to something like MusicGen or Riffusion in terms of coherence across entire compositions?
- nprateem 1y ago
  They always fail with structure. The progressions often meander aimlessly and eventually go to weird places, at least IME.
gitroom 1y ago
Man, this whole topic hits way harder than I expected. AI taking shots at music creation makes me feel a bit hyped but also kinda iffy, especially when I hear people say it plays it way too safe. You think keeping things safe in AI art helps anyone actually level up or just holds us all back?
- phatskat 1y ago
  If the “Guns in butts” / “my wife is a jar of dirt” songs are any indication, I don’t know how “safe” it will be, at least as far as content goes. I’m similarly hyped and iffy. If you could have a model that listens to a looping segment to contextualize it, and then play with other patterns on top but through a more expressive way (or even humming/singing and allowing the LLM of sorts to compose it together), that could be interesting. Would it be panned for being AI-assisted? I’d hope not, I think? “I hear you're buying a synthesizer and an arpeggiator and are throwing your computer out the window because you want to make something real. You want to make a Yaz record. I hear that you and your band have sold your guitars and bought turntables. I hear that you and your band have sold your turntables and bought guitars.” We just add AI in whatever forms it takes to the list I suppose.
briga 1y ago
Interesting how there is no mention of how the training data for this was collected. This does sound quite a bit better than Meta's MusicGen, but then again that model was also trained on a small licensed dataset.
- platers 1y ago
  It sounds very similar to suno v3.5 (including the audio quality) Likely they trained on suno generations.
Semaphor 1y ago
> aggressive, Heavy Riffs, Blast Beats, Satanic Black Metal Result: A generic pop-rock song without riffs or blast beats. Not even power metal or corset core, let alone anything even slightly resembling Black Metal. Yup. Still doing what I expect from AI music.
everdrive 1y ago
Yes, please ruin music. Ruin everything you can. As long as you can build it, you should ruin it. There's really no limit. It's the masses who will actually do the ruining, so those building the technology are totally blameless. And you might even make some money, so it's all worth it.
- lc9er 1y ago
  These music/art AI threads are always gross. Make things for the joy of creating. Prompting some AI model doesn’t make you a musician - it actively robs you of the thing that’s rewarding about actually creating something.
  - awwaiid 1y ago
    Some get joy from creating generative tools/models!
  - gavinray 1y ago
    Have you considered that some people produce music solely for aesthetic purposes? The main genre of music I listen to is electronic. Many electronic songs are written to evoke a specific feeling, without meaningful lyrics. When I produce electronic music, I have no particular lyrics or composition in mind. I just fiddle around with different sound layers going for a particular "vibe" and mash things together until they sound good to me. I see AI as another tool to expedite this process.
    - lc9er 1y ago
      Sure. If your music means nothing to you, than by all means, make it with methods that remove all agency from the process. I’ve written instrumental/electronic music, too. When I do, it means something to me. The last thing I want is for an AI to make it for me.
- ronsor 1y ago
  A tool existing doesn't ruin anything. Genuinely stop being so dramatic and learn to ignore things you don't like. Our society would be a lot better and calmer if people did that rather than start pointless crusades.

109 points by wertyk 1y ago

frankfrank13 1y ago

Fun to play with!

hammock 1y ago

Is there a demo hosted somewhere?

frankfrank13 1y ago
- https://ace-step.github.io/ https://ace-step.github.io/ - https://huggingface.co/spaces/ACE-Step/ACE-Step https://huggingface.co/spaces/ACE-Step/ACE-Step

Rodeoclash 1y ago

As a musician, the things I want most from generative AI is: 1. Being able to have the AI fill in a track in the song, but use the whole song as input to figure out what to generate. Ideally for drums this would be a combination of individual drum hits, effects and midi so I'm able to tweak it after generation. If it used the Ableton effects and drum rack then that would be perfect. 2. Take my singing and make it both sound great and like any combination of great singers (e.g. give me a bit of Taylor Swift combined with Cat Power) I've had a play with the style transfer between singers (bullet point 2 above) but when I last tried it, it was garbage in / garbage out, and my singing is garbage. What I don't want: To just generate a whole song. Adobe does this style of assistive AI well in the photo editing space but no one seems to have brought it to audio yet.

pelagic_sky 1y ago
Exactly. I don’t want AI to make me a song. I want it to switch up my baseline or chords or recommend a fill instrument.
vunderba 1y ago
Logic Pro comes the closest to this with the addition of virtual drummers. You can assign it to follow certain rhythmic timings by connecting it to a main instrument track, and by labeling sections of your song (bridge, chorus, etc) you can regenerate until you find something you're happy with. It's a far cry from having a real drummer, but it works in a pinch.
viraptor 1y ago
Spark amp is now advertising with some interesting AI options. https://au.positivegrid.com/products/spark-2 https://au.positivegrid.com/products/spark-2 As far as I understand it can attempt to generate backing drums (and music?) for jamming, which sounds great for practicing.
dstorm1 1y ago
I think you might like https://soundry.ai https://soundry.ai they are working on a feature that is just like 1.
nprateem 1y ago
I think the big limiting factor is having it learn what the infinite number of parameters in different synths, etc actually make it sound like, and then being able to make the model be able to produce music by using and configuring them instead of just spaffing out crap music. If it could do that it'd give the tweakability it needs, but I think it would basically involve training a model for each instrument.

architectonic 1y ago

How do the quality and prompt adherence compare to Suno v4?

999900000999 1y ago

> RapMachine Fine-tuned on pure rap data to create an AI system specialized in rap generation Expected capabilities include AI rap battles and narrative expression through rap Rap has exceptional storytelling and expressive capabilities, offering extraordinary application potential Using a certain other music generator I got it to accidentally say ***. It said it with a Latino American accent too. In fact for whatever reason this tool couldn’t use a typical AAVE voice. Just Sage Francis / Atmosphere like dictionary raps and a few Latino American ones. A big limitation of AI sloop is it tries to not offend anyone. Art that can’t even try to offend is barely art.

Duwensatzaj 1y ago
> Art that can’t even try to offend is barely art. Good art is secondary to avoiding some journalist writing a hit piece about how they used your company’s AI generator to depict Hitler saying the gamer word.
- 999900000999 1y ago
  AI generated articles about ai generated music which is generated from reading ai generated articles. Shall it continue in an unholy loop until the end of time ?
- lwansbrough 1y ago
  To be fair, nothing AI generated is inherently “good art” if you take artistic intent into account.
  - justlikereddit 1y ago
    Mixing paint in your rectum with the intention of making good art will always rate lower than AI slop
    - nxpnsv 1y ago
      *citation needed

dheera 1y ago

The diagram is super vague. How are the lyrics encoded? What does the encoder look like inside? What is the input size, input format, output size, output format? Are the three encoder outputs added? Concatentated? When Mert and m-Hubert combine are they added? Multiplied? Subtracted? Concatenated? I really wish people could make better diagrams.

lilcrise 1y ago

VPS 4 Cores SSD Disk Space: 200 GB CPU cores: 4 RAM: 4 GB Ubuntu Server 22.04 can it run on vps server this small?

montag 1y ago

The experimental section was most interesting: https://ace-step.github.io/#Experimental https://ace-step.github.io/#Experimental Puh tss puh tss Kshhh ff kshhh ff Bmm tkk bmm tkk Drr-dr-d-d-dt

l72 1y ago

I want to play something on my keyboard (the only instrument I am slightly ok at) and then be able to tell it to play it with a saxophone and describe exactly how I want it played. I don’t need an AI to create a song for me, I need 100 session musicians at my disposal to create the song I want. I am very excited about having that type of ai.

spyder 1y ago
Here is a cool demonstration to do voice-to-instrument or instrument-to-another instrument (The inconvenient thing is that for a new kind of output sound you have to train a model for around 1 hour for good quality, but after that you can use it with different inputs quickly): https://youtu.be/lI1LCfTx2lI?t=525 https://youtu.be/lI1LCfTx2lI?t=525 There is also Kits.ai https://www.kits.ai/tools/ai-instruments https://www.kits.ai/tools/ai-instruments
arafalov 1y ago
You sound like you are just talking about virtual instruments, which are getting better and better. Something like this maybe, doing the virtual instruments (VST/VSTI) review for saxophone: https://news.ycombinator.com/item?id=43909398 https://news.ycombinator.com/item?id=43909398

peterzhang2000 1y ago

[dead]

Alex_001 1y ago

Really interesting — we're seeing more efforts now to bring the "foundation model" approach to creative domains like music, but I wonder how well these models can internalize musical structure over long time scales. Has anyone here compared ACE-Step to something like MusicGen or Riffusion in terms of coherence across entire compositions?

nprateem 1y ago
They always fail with structure. The progressions often meander aimlessly and eventually go to weird places, at least IME.

gitroom 1y ago

Man, this whole topic hits way harder than I expected. AI taking shots at music creation makes me feel a bit hyped but also kinda iffy, especially when I hear people say it plays it way too safe. You think keeping things safe in AI art helps anyone actually level up or just holds us all back?

phatskat 1y ago
If the “Guns in butts” / “my wife is a jar of dirt” songs are any indication, I don’t know how “safe” it will be, at least as far as content goes. I’m similarly hyped and iffy. If you could have a model that listens to a looping segment to contextualize it, and then play with other patterns on top but through a more expressive way (or even humming/singing and allowing the LLM of sorts to compose it together), that could be interesting. Would it be panned for being AI-assisted? I’d hope not, I think? “I hear you're buying a synthesizer and an arpeggiator and are throwing your computer out the window because you want to make something real. You want to make a Yaz record. I hear that you and your band have sold your guitars and bought turntables. I hear that you and your band have sold your turntables and bought guitars.” We just add AI in whatever forms it takes to the list I suppose.

briga 1y ago

Interesting how there is no mention of how the training data for this was collected. This does sound quite a bit better than Meta's MusicGen, but then again that model was also trained on a small licensed dataset.

platers 1y ago
It sounds very similar to suno v3.5 (including the audio quality) Likely they trained on suno generations.

Semaphor 1y ago

> aggressive, Heavy Riffs, Blast Beats, Satanic Black Metal Result: A generic pop-rock song without riffs or blast beats. Not even power metal or corset core, let alone anything even slightly resembling Black Metal. Yup. Still doing what I expect from AI music.

everdrive 1y ago

Yes, please ruin music. Ruin everything you can. As long as you can build it, you should ruin it. There's really no limit. It's the masses who will actually do the ruining, so those building the technology are totally blameless. And you might even make some money, so it's all worth it.

lc9er 1y ago
These music/art AI threads are always gross. Make things for the joy of creating. Prompting some AI model doesn’t make you a musician - it actively robs you of the thing that’s rewarding about actually creating something.
- awwaiid 1y ago
  Some get joy from creating generative tools/models!
- gavinray 1y ago
  Have you considered that some people produce music solely for aesthetic purposes? The main genre of music I listen to is electronic. Many electronic songs are written to evoke a specific feeling, without meaningful lyrics. When I produce electronic music, I have no particular lyrics or composition in mind. I just fiddle around with different sound layers going for a particular "vibe" and mash things together until they sound good to me. I see AI as another tool to expedite this process.
  - lc9er 1y ago
    Sure. If your music means nothing to you, than by all means, make it with methods that remove all agency from the process. I’ve written instrumental/electronic music, too. When I do, it means something to me. The last thing I want is for an AI to make it for me.
ronsor 1y ago
A tool existing doesn't ruin anything. Genuinely stop being so dramatic and learn to ignore things you don't like. Our society would be a lot better and calmer if people did that rather than start pointless crusades.