5 ms·
I want to do the reverse: pass in some music and have it describe what I'm hearing
by dyno12345 4y ago
I want to do the reverse: pass in some music and have it describe what I'm hearing
- brookst 4y agoYes! And not just notes or chords or time signatures, but real analysis of how the piece works.
- deleted 4y ago[deleted]
- MadDemon 4y agohttps://samplab.com https://samplab.com is doing something like that.
- dimatura 4y agoI believe parts of this model could be adapted to accomplish this. The "mulan" network generates embeddings from music and text so that they "match" each other, i.e. audio segments and text captions that go well together would be mapped to the same embedding. At run-time, they use embeddings from the text to generate the music. So conceivably the opposite could be done, using the "mulan" embeddings from audio only to generate text, by replacing the "soundstream" decoder with a text LLM. That said, there's probably simpler methods that could generate decent results, and I'd be surprised if there isn't already some out there. It's worth noting that along with this work they released a dataset of music with captions that could also help training new models for this task.
- ilaria_m 4y agoMusCaps: Generating Captions for Music Audio: https://arxiv.org/abs/2104.11984 https://arxiv.org/abs/2104.11984
- 52-6F-62 4y agoI think you’re missing the point. Music is an experience, not a checkbox or text book needing summarized.
- metaketa 4y agoWorking on the same dream: https://parture.org https://parture.org