4 ms·
Music ControlNet: Multiple Time-Varying Controls for Music Generation
- GaggiX 3y agoThe model used here is very small, 41M, I wonder how well it would scale at a bigger size.
- TaylorAlexander 3y agoI was thinking recently, now that we have multimodal text and image models, music and sound generation will probably get rolled in to the big foundation models. And then we can look at adding more niche modalities like 3D model generation. As we begin to explore large numbers of modalities we will have highly generalized models.
- bongwater_OS 3y agoLove seeing the MIR research from CMU recently. Chris Donahue is the man!!
- brrrrrm 3y agocan I try this out somewhere?
- SpaceManNabs 3y agoI understand that this paper is about controls. I wish there was more detail in how it differs to other music generation methods like MusicLM. That seems to be in the MusicGen paper though [5]! But then I am more curious about how this compares to MusicLM in terms of music generation.
- comex 3y agoLooking at the "Melody & Rhythm Control" section under "Cherry-picked"... the rhythm control is weird. In many of the examples, the generated music clearly has a different BPM from the reference. But the model seems to still be trying to align the notes in units of time (rather than beats), so the notes get desynced with the beat. The model then tries to cover up the discrepancy and make it sound like syncopation, by emphasizing or de-emphasizing or outright altering notes, but it doesn't work very well. Maybe conditioning on the BPM would help?
- blurbleblurble 3y agoAwesome, I've been curious about using controlnet like this. I'm glad someone tried it out.