3 ms·

Has anyone solved the data problem? I have taken many stabs at this over the last 10 years, and the primary problem is that music is _extremely_ subtle. Even so

Has anyone solved the data problem? I have taken many stabs at this over the last 10 years, and the primary problem is that music is _extremely_ subtle. Even something as seemingly simple as chord progressions are very difficult to annotate properly without deep training. And even then, experts can disagree. The thing that's missing is the data. If we had midi transcriptions of 100k songs (abc notation could be fine too), we could probably get really interesting stuff, but most of what is available is lossy chord transcriptions and classical music (public domain). So if you want to automatically create something that sounds like mozart, you're in luck! But this isn't really satisfying to me. For generative music, we're still largely stuck with encoding musical rules in code rather than feeding data to a transformer. To me, the former feels much less like AI than the latter. The data is all locked up behind an impossible quagmire of copyright. But if I were a sheet music publishing company, I would be seriously considering the future of music creation with AI given my broad access to notated music & metadata (is this an original score, or a grade 1 simplification?). But again, music copyright is a pretty complex contraption.

songeater 4y ago
>> has anyone solved the data problem? So MIDI songs were being generated quite "early" on, even with basic text generators (LSTM models), where the "alphabet" was replaced by MIDI symbols. [1] What Jukebox (and other models) did well was work with raw audio, rather than rely on MIDI or similar transcriptions. They break down audio into "blocks" using Fast Fourier Transforms, then each block is "tokenized" and then fed into transformers, similar to GPT-3 or other newer text generation models. This now allows the "subtle" musicalities to be discovered by models without relying on transcription. Wavenet (from Google) [2] I believe was the first one to do this... I tried myself for 6 months, but the processing power required for this is candidly only available for the Google/OpenAIs of the world. >> the data is all locked up behind an impossible quagmire of copyright So my understanding is that the training of Jukebox is similar to Copilot. It was trained on the entire English Spotify catalogue so all that has been digested by the model - under Fair Use. But again - this is probably far from "settled" legally. [1] https://arxiv.org/abs/1612.07837 https://arxiv.org/abs/1612.07837 [2] https://www.deepmind.com/blog/wavenet-a-generative-model-for-raw-audio https://www.deepmind.com/blog/wavenet-a-generative-model-for...

songeater 4y ago

>> has anyone solved the data problem? So MIDI songs were being generated quite "early" on, even with basic text generators (LSTM models), where the "alphabet" was replaced by MIDI symbols. [1] What Jukebox (and other models) did well was work with raw audio, rather than rely on MIDI or similar transcriptions. They break down audio into "blocks" using Fast Fourier Transforms, then each block is "tokenized" and then fed into transformers, similar to GPT-3 or other newer text generation models. This now allows the "subtle" musicalities to be discovered by models without relying on transcription. Wavenet (from Google) [2] I believe was the first one to do this... I tried myself for 6 months, but the processing power required for this is candidly only available for the Google/OpenAIs of the world. >> the data is all locked up behind an impossible quagmire of copyright So my understanding is that the training of Jukebox is similar to Copilot. It was trained on the entire English Spotify catalogue so all that has been digested by the model - under Fair Use. But again - this is probably far from "settled" legally. [1] https://arxiv.org/abs/1612.07837 https://arxiv.org/abs/1612.07837 [2] https://www.deepmind.com/blog/wavenet-a-generative-model-for-raw-audio https://www.deepmind.com/blog/wavenet-a-generative-model-for...