23 ms·
>"More recently, there is a shift towards using a Transformer architecture, and right now I’m experimenting with that as well." I'm really curious- any early r
by citnaj 8y ago
>"More recently, there is a shift towards using a Transformer architecture, and right now I’m experimenting with that as well."
I'm really curious- any early results to share on that? Attention really does make a big difference on a lot of things (including work I've done so I know first hand). It should improve the coherence of the entire music piece in theory at least, right?
- minimaxir 8y agotextgenrnn (https://github.com/minimaxir/textgenrnn https://github.com/minimaxir/textgenrnn) uses a simple Attention Weighted Average at the end of the model for text generation, which in my testing allows the model to learn much better.
- mcleavey 8y agoTransformer is working really well- I'm very excited. I'll probably be sharing results soon. Yes, the attention makes a huge difference & the pieces are both more creative and more coherent.
- citnaj 8y agoThat is exciting! I'll be watching on Twitter :)
- gwern 8y agoHave you considered using 'learning from human preferences' as the loss function in addition to the Transformers? That was another OpenAI project, and it seems tailor-made for music generation: what is more 'I know it when I hear it' than music quality?
- kasrahbar 8y agoCheck out Music Transformer that was recently published https://arxiv.org/abs/1809.04281 https://arxiv.org/abs/1809.04281 Some generated samples: https://storage.googleapis.com/music-transformer/index.html https://storage.googleapis.com/music-transformer/index.html
- bcheung 8y agoThe accompaniment examples are cool. That would be a very nice tool to have. Just play a melody and it auto-generates an accompaniment.