5 ms·

>> In fact you could imagine a musical version of this where the operator is describing feelings or styles and the AI is generating music to match where the ope

>> In fact you could imagine a musical version of this where the operator is describing feelings or styles and the AI is generating music to match where the operator wants to go. OpenAI already did this with Jukebox! I'm just surprised more people have not used it as an "instrument" vs as a novelty. shameless plug: https://soundcloud.com/songshtr/sets/2106-ep https://soundcloud.com/songshtr/sets/2106-ep

smthngwitty 5y ago
One downside with Jukebox: it takes ~9 hours to generate ~90 seconds of audio (even on a NVIDIA V100 GPU) since it's an auto-regressive model making experimentation and 'co-creation' much harder
- songeater 5y ago
  agreed. full waveform "music" (unlike say MIDI) is just many more variables than a "picture" or even "video". Also you have to "stitch together" a lot of these samples to get anything that resembles a "new" song. More akin to mining... but still its kind of crazy what can be done with it.
  - echelon 5y ago
    I'm not familiar with Jukebox, so maybe I'm wrong. It seems like you'd want a fast architecture to use a model to generate sheet music / MIDI, then layer another model on top to create instrumentals?
  - songeater 5y ago
    >> Why doesn't the architecture use a model to generate sheet music / MIDI, then layer another model on top to create instrumentals? there are various models that do this [0]... just doesn't have the same power to generate waveform music that jukebox does. Jukebox effectively is a 2 step model a) a 3 layer VQ-VAE compresses the music (so think of this as the MIDI equivalent) and b) a transfomer then learns/generates sequences. The compression is the expensive part of the model. [0] https://towardsdatascience.com/how-to-generate-music-using-a-lstm-neural-network-in-keras-68786834d4c5 https://towardsdatascience.com/how-to-generate-music-using-a...
- songeater 5y ago
  >> it takes ~9 hours to generate ~90 seconds of audio (even on a NVIDIA V100 GPU) btw this is true if you upsample all 3 levels. I have found in practise that you are fine upsampling 2 levels and then using a DAW to "clean up"/"remaster" [0]... again will work for certain sounds better than others. The last upsampling step is by far the most expensive. so cutting that down cuts total time by 75% [0] simple hacks such as using a low-pass filter

smthngwitty 5y ago

One downside with Jukebox: it takes ~9 hours to generate ~90 seconds of audio (even on a NVIDIA V100 GPU) since it's an auto-regressive model making experimentation and 'co-creation' much harder

songeater 5y ago
agreed. full waveform "music" (unlike say MIDI) is just many more variables than a "picture" or even "video". Also you have to "stitch together" a lot of these samples to get anything that resembles a "new" song. More akin to mining... but still its kind of crazy what can be done with it.
- echelon 5y ago
  I'm not familiar with Jukebox, so maybe I'm wrong. It seems like you'd want a fast architecture to use a model to generate sheet music / MIDI, then layer another model on top to create instrumentals?
- songeater 5y ago
  >> Why doesn't the architecture use a model to generate sheet music / MIDI, then layer another model on top to create instrumentals? there are various models that do this [0]... just doesn't have the same power to generate waveform music that jukebox does. Jukebox effectively is a 2 step model a) a 3 layer VQ-VAE compresses the music (so think of this as the MIDI equivalent) and b) a transfomer then learns/generates sequences. The compression is the expensive part of the model. [0] https://towardsdatascience.com/how-to-generate-music-using-a-lstm-neural-network-in-keras-68786834d4c5 https://towardsdatascience.com/how-to-generate-music-using-a...
songeater 5y ago
>> it takes ~9 hours to generate ~90 seconds of audio (even on a NVIDIA V100 GPU) btw this is true if you upsample all 3 levels. I have found in practise that you are fine upsampling 2 levels and then using a DAW to "clean up"/"remaster" [0]... again will work for certain sounds better than others. The last upsampling step is by far the most expensive. so cutting that down cuts total time by 75% [0] simple hacks such as using a low-pass filter