4 ms·
Deep learning needs a lot if training data. There’s not enough MIDI encoded music to train something like a GPT-3 model to generate MIDI sequences. A better way
by p1esk 5y ago
Deep learning needs a lot if training data. There’s not enough MIDI encoded music to train something like a GPT-3 model to generate MIDI sequences. A better way is to train on and generate raw audio, see OpenAI JukeBox. Unfortunately it’s extremely compute intensive (even compared to GPT-3), so it will probably be a few more years until they (or some other big player) releases JukeBox-2.
- zozbot234 5y agoThe amount of data you need depends on the model architecture you're using. A generic model like GPT-3 is neither here nor there, but something specifically intended for music can make do with very little data.
- p1esk 5y agoWhat do you mean “neither here nor there”?
- zozbot234 5y agoThe structure of something like GPT-3 is far too weak and general to achieve good results for something as structurally complex as music. It's designed to generate text - and then mostly natural language text. Music is very different, as OP hints in the linked post.
- p1esk 5y agoI thought you wanted to use deep learning. In DL transformers are the best we got currently. Both JukeBox and MuseNet use transformers. What makes you think they are not up to the task?
- zozbot234 5y ago"Transformers" is a general technique, comparable to "LSTM" or "attention". The details of just how much prior information about the domain you reflect in the model architecture are just as relevant, and it's not clear just how well MuseNet addresses this compared to the earlier work I mentioned above. As for JukeBox, it's trying to solve audio generation at the raw samples level, which is quite literally a complexity increase of several orders of magnitude.
- p1esk 5y agoA transformer architecture does not change whether you use word tokens, musical chord encodings, audio waveform samples, image patches, or video frames. You can change a number of layers or attention heads, but the architecture remains the same - it does not care what is the nature of sequences it encodes/decodes. That's kind of the point - to have a universal algorithm. Human neocortex does not care which sensory receptors the signals are coming from, there's evidence that it processes all of them in the same manner. Any prior information about the domain can be encoded in a way you prepare the input samples, and this would be equally useful and relevant regardless of the DL architecture you decide to use (RNNs, autoencoders, GANs, transformers). Using raw audio samples has two advantages: First, you provide all available training data to a model, and trust that it will learn to extract what it needs from it. Second, unlike any other music representation formats, you have huge amounts of training data. So essentially all you need is a huge transformer and many thousands of GPUs to train it on, using all mp3s you can get your hands on, and you can reasonably expect a similar level of quality as the quality of text generated by GPT-3. What do you think?
- zozbot234 5y ago> Any prior information about the domain can be encoded in a way you prepare the input samples Adding features to provide further information about the input data can be helpful, but not nearly as much as ensuring that strong priors are reflected in the actual model architecture, whenever relevant to the domain. Johnson's paper about biaxial RNN provides one example of how to do this in the context of music.