4 ms·
Could such a AI be reversed, as in basically become a stable diffusion generation of music, with just keywords and a noise generator?
by qikInNdOutReply 4y ago
Could such a AI be reversed, as in basically become a stable diffusion generation of music, with just keywords and a noise generator?
- rzzzt 4y agoJukebox is using a similar scheme: https://openai.com/blog/jukebox/#approach https://openai.com/blog/jukebox/#approach
- qikInNdOutReply 4y agoThanks.
- aljungberg 4y agoYes, I believe it had been done with Google’s Wavenet for example where normally it’s trained to generates speech conditioned on textual input. For fun they trained it on piano music and gave it no conditioning and it improvised piano pieces. Also see OpenAI’s jukebox.
- songeater 4y agoSo Jukebox can actually be broken down into 2 components / 3 steps: A) compression (take 44khz audio and compress into tokens using 3 VQVAE layers) B) generation (take the compressed tokens and use that on a transformer to generate sequences of those tokens) C) de-compresss (reverse the VQVAE) -> this is BY FAR the longest/most computationally expensive step Diffusion currently wont really help with (B) or even (A)... but there is a lot of new experiments going on with (3)... where the decompression using stable-diffusion is providing very interesting results using far less compute... check out JuicyJukebox notebooks (link to come... cant access github colab from my work computer).