5 ms·

One downside with Jukebox: it takes ~9 hours to generate ~90 seconds of audio (even on a NVIDIA V100 GPU) since it's an auto-regressive model making experimenta

One downside with Jukebox: it takes ~9 hours to generate ~90 seconds of audio (even on a NVIDIA V100 GPU) since it's an auto-regressive model making experimentation and 'co-creation' much harder

songeater 5y ago
agreed. full waveform "music" (unlike say MIDI) is just many more variables than a "picture" or even "video". Also you have to "stitch together" a lot of these samples to get anything that resembles a "new" song. More akin to mining... but still its kind of crazy what can be done with it.
- echelon 5y ago
  I'm not familiar with Jukebox, so maybe I'm wrong. It seems like you'd want a fast architecture to use a model to generate sheet music / MIDI, then layer another model on top to create instrumentals?
- songeater 5y ago
  >> Why doesn't the architecture use a model to generate sheet music / MIDI, then layer another model on top to create instrumentals? there are various models that do this [0]... just doesn't have the same power to generate waveform music that jukebox does. Jukebox effectively is a 2 step model a) a 3 layer VQ-VAE compresses the music (so think of this as the MIDI equivalent) and b) a transfomer then learns/generates sequences. The compression is the expensive part of the model. [0] https://towardsdatascience.com/how-to-generate-music-using-a-lstm-neural-network-in-keras-68786834d4c5 https://towardsdatascience.com/how-to-generate-music-using-a...
songeater 5y ago
>> it takes ~9 hours to generate ~90 seconds of audio (even on a NVIDIA V100 GPU) btw this is true if you upsample all 3 levels. I have found in practise that you are fine upsampling 2 levels and then using a DAW to "clean up"/"remaster" [0]... again will work for certain sounds better than others. The last upsampling step is by far the most expensive. so cutting that down cuts total time by 75% [0] simple hacks such as using a low-pass filter

One downside with Jukebox: it takes ~9 hours to generate ~90 seconds of audio (even on a NVIDIA V100 GPU) since it's an auto-regressive model making experimentation and 'co-creation' much harder

songeater 5y ago

agreed. full waveform "music" (unlike say MIDI) is just many more variables than a "picture" or even "video". Also you have to "stitch together" a lot of these samples to get anything that resembles a "new" song. More akin to mining... but still its kind of crazy what can be done with it.

echelon 5y ago
I'm not familiar with Jukebox, so maybe I'm wrong. It seems like you'd want a fast architecture to use a model to generate sheet music / MIDI, then layer another model on top to create instrumentals?
songeater 5y ago
>> Why doesn't the architecture use a model to generate sheet music / MIDI, then layer another model on top to create instrumentals? there are various models that do this [0]... just doesn't have the same power to generate waveform music that jukebox does. Jukebox effectively is a 2 step model a) a 3 layer VQ-VAE compresses the music (so think of this as the MIDI equivalent) and b) a transfomer then learns/generates sequences. The compression is the expensive part of the model. [0] https://towardsdatascience.com/how-to-generate-music-using-a-lstm-neural-network-in-keras-68786834d4c5 https://towardsdatascience.com/how-to-generate-music-using-a...

songeater 5y ago

>> it takes ~9 hours to generate ~90 seconds of audio (even on a NVIDIA V100 GPU) btw this is true if you upsample all 3 levels. I have found in practise that you are fine upsampling 2 levels and then using a DAW to "clean up"/"remaster" [0]... again will work for certain sounds better than others. The last upsampling step is by far the most expensive. so cutting that down cuts total time by 75% [0] simple hacks such as using a low-pass filter