4 ms·

I previously tried an approach which uses DRL for feedback but I couldn't quite get it to work: https://www.gwern.net/GPT-2-preference-learning https://www.gwer

by gwern 7y ago

I previously tried an approach which uses DRL for feedback but I couldn't quite get it to work: https://www.gwern.net/GPT-2-preference-learning https://www.gwern.net/GPT-2-preference-learning At the moment, it probably would be more practical to train a model to predict ratings and use that to screen generated samples or possible completions and throw out too-low-scoring ones (the 'ranker' approach worked out very well for the Meena chatbot recently: https://arxiv.org/abs/2001.09977 https://arxiv.org/abs/2001.09977 )

p1esk 7y ago
What do you think about training a binary classifier to distinguish between human and generated samples? E.g. choose a state of the art model designed to classify composers or styles, and finetune it for this task.
- mattkrause 7y ago
  That's a tiny step away from a GAN, which works very well for image generation, so seems promising here too.
  - p1esk 7y ago
    GANs have been tried many times already for music generation, without much success. GPT-2 works very well for text generation so it seemed promising here too. Music falls somewhere in between text (as a sequence of chords or PCM samples) and image (as a piano roll or a spectrogram), so maybe some hybrid of image and text generators is needed.
    - TheOtherHobbes 7y ago
      Music does neither, which is why these naive approaches don't work.
- gwern 7y ago
  I think that could work potentially as a rejection sampling, but you also have the risk that it will simply find some small discriminative detail and be unusefully good at the classification; that's why you do it in a loop as a GAN, but as you mention, GANs work really badly on sequences, still, so... If you wanted to improve my ABC-MIDI GPT-2, the most straightforward ways would be to do data cleaning (I'm sure there's tens of thousands of awful MIDI files which should be removed! data cleaning with RNNs or GPT-2 or GANs always makes a large difference) and increase the model size (the fact that loss bottomed at 0.20, which is still quite bad, suggests that MIDI is hard enough that GPT-2 is struggling). More interesting would be to use Reformer or another long-range Transformer and try to operate directly on a more raw representation, like the the piano roll representation of MIDI. I think GPT-2 makes a lot of syntax errors which cripple outputs when a 'voice' goes silent, and a piano roll representation would be a lot more robust (at the cost of being like 10x larger).
  - p1esk 7y ago
    How would you present a piano roll to a transformer (e.g. what would be a sample of the sequence)? You could try using a tuple of pitch integers for each time step. I'm not sure how big a "vocabulary" would need to be to capture most of the chords (note combinations) - it might actually be comparable in size to a language vocabulary (tens of thousands of words). You could use two channels to capture note onset/offset info (like it was done in a biaxial RNN paper). Or the encoding used for Musenet (with explicit timing info), but somehow I like the idea of "chords as words" better.
    - gwern 7y ago
      You would simply have a vector of 128 for each timestep (I'm not sure how discrete MIDI goes), and any of the 128 instruments turned on would be non-zero. It's not a dense encoding, by a long shot, but it's quite literal and explicit so you don't have to worry indirectness making it hard to learn (assuming you have a model which can handle such long-range inputs to begin with). Personally, I'd rather try redoing BPE encoding for ABC-MIDI specifically.

by gwern 7y ago

p1esk 7y ago

What do you think about training a binary classifier to distinguish between human and generated samples? E.g. choose a state of the art model designed to classify composers or styles, and finetune it for this task.

mattkrause 7y ago
That's a tiny step away from a GAN, which works very well for image generation, so seems promising here too.
- p1esk 7y ago
  GANs have been tried many times already for music generation, without much success. GPT-2 works very well for text generation so it seemed promising here too. Music falls somewhere in between text (as a sequence of chords or PCM samples) and image (as a piano roll or a spectrogram), so maybe some hybrid of image and text generators is needed.
  - TheOtherHobbes 7y ago
    Music does neither, which is why these naive approaches don't work.
gwern 7y ago
I think that could work potentially as a rejection sampling, but you also have the risk that it will simply find some small discriminative detail and be unusefully good at the classification; that's why you do it in a loop as a GAN, but as you mention, GANs work really badly on sequences, still, so... If you wanted to improve my ABC-MIDI GPT-2, the most straightforward ways would be to do data cleaning (I'm sure there's tens of thousands of awful MIDI files which should be removed! data cleaning with RNNs or GPT-2 or GANs always makes a large difference) and increase the model size (the fact that loss bottomed at 0.20, which is still quite bad, suggests that MIDI is hard enough that GPT-2 is struggling). More interesting would be to use Reformer or another long-range Transformer and try to operate directly on a more raw representation, like the the piano roll representation of MIDI. I think GPT-2 makes a lot of syntax errors which cripple outputs when a 'voice' goes silent, and a piano roll representation would be a lot more robust (at the cost of being like 10x larger).
- p1esk 7y ago
  How would you present a piano roll to a transformer (e.g. what would be a sample of the sequence)? You could try using a tuple of pitch integers for each time step. I'm not sure how big a "vocabulary" would need to be to capture most of the chords (note combinations) - it might actually be comparable in size to a language vocabulary (tens of thousands of words). You could use two channels to capture note onset/offset info (like it was done in a biaxial RNN paper). Or the encoding used for Musenet (with explicit timing info), but somehow I like the idea of "chords as words" better.
  - gwern 7y ago
    You would simply have a vector of 128 for each timestep (I'm not sure how discrete MIDI goes), and any of the 128 instruments turned on would be non-zero. It's not a dense encoding, by a long shot, but it's quite literal and explicit so you don't have to worry indirectness making it hard to learn (assuming you have a model which can handle such long-range inputs to begin with). Personally, I'd rather try redoing BPE encoding for ABC-MIDI specifically.