5 ms·
If you are into this kind of stuff (time series density/probability modeling), there are several places to go beyond this. The simplest example is the RNN-RBM
by kastnerkyle 11y ago
If you are into this kind of stuff (time series density/probability modeling), there are several places to go beyond this.
The simplest example is the RNN-RBM (restricted boltzmann machine) from the Theano deep learning tutorials [1]. There are follow-up papers by Kratarth Goel extending this to LSTM-DBN, and RNN-NADE (neural autoregressive density estimator) has also been explored in this space. The results for MIDI are pretty good.
Ales Graves [2] has done a huge amount of work in sequence modeling for classification and generation. His work with LSTM-GMM for online handwriting modeling can also be applied to speech features (vocoded speech), and seems to work well. No official publication to date on the speech experiments themselves, but the procedure is well documented in his paper [3]
The most recent thing is our paper "A Recurrent Latent Variable Model for Sequential Data" [4]. It operates on raw timeseries (!) even though all intuition would say that an RNN shouldn't handle this well. Adding a recurrent prior seems to preserve style in the handwriting generations at least - I may try on this "Let It Go!" task and see what happens! We are also exploring conditional controls and higher-level speech features, which should result in samples as good or better than LSTM-GMM.
[1] http://deeplearning.net/tutorial/rnnrbm.html#rnnrbm http://deeplearning.net/tutorial/rnnrbm.html#rnnrbm
[2] https://www.youtube.com/watch?v=-yX1SYeDHbg#t=36m50s https://www.youtube.com/watch?v=-yX1SYeDHbg#t=36m50s
[3] http://arxiv.org/abs/1308.0850 http://arxiv.org/abs/1308.0850
[4] http://arxiv.org/abs/1506.02216 http://arxiv.org/abs/1506.02216
- kastnerkyle 11y agoSpecifically, for these kinds of sequences (chord sequences, etc.) check out models that uses note-level features but with joint probability (RNN with sigmoid out and cross-entropy loss, RBM, NADE, etc.). It should do even better!
- thanatropism 11y agoSo I took a class in college (15 years ago) where we learned how to do some backprop (out of plain Matlab). I've been seeing impressive demonstrations of RNNs these weeks. How (if at all) can a Matlab-Fortran-SAS grunt start-from-the-beginning to at least understand the rudiments of how these work?
- kastnerkyle 11y agoThere is a nice book by my profs (Y. Bengio, A. Courville) and a former student of the lab (I. Goodfellow) here: http://www.iro.umontreal.ca/~bengioy/DLbook/ http://www.iro.umontreal.ca/~bengioy/DLbook/ The chapter on RNNs is pretty enlightening. You probably want to start with something the goes through feedforward networks, convolutional, etc. first though. Andrew Ng's Coursera course does a super basic NN in MATLAB which might be good to shake the rust off [1]. Hugo Larochelle's youtube course [2], Geoff Hinton's Coursera course [3], and Nando's youtube Deep Learning course [4] are all very good. There are also some great lectures from Aaron Courville from the Representation Learning class that just finished here at UdeM [5] If you want to do this for real (not self teaching RNN backprop on toy examples) tool-wise you generally go with either Theano (my preference) or Torch - implementing the backwards path for these (and verifying gradients are correct) is not pretty. Theano does it almost for free, and Torch can do it at the module level. Either way you definitely want one of the higher order tools! [1] https://www.coursera.org/learn/machine-learning https://www.coursera.org/learn/machine-learning [2] https://www.youtube.com/watch?v=SGZ6BttHMPw&list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH https://www.youtube.com/watch?v=SGZ6BttHMPw&list=PL6Xpj9I5qX... [3] https://www.coursera.org/course/neuralnets https://www.coursera.org/course/neuralnets [4] https://www.youtube.com/watch?v=PlhFWT7vAEw https://www.youtube.com/watch?v=PlhFWT7vAEw [5] https://ift6266h15.wordpress.com/ https://ift6266h15.wordpress.com/
- thanatropism 11y agoThanks for the references!
- LukeB_UK 11y agoThe infinite jukebox[0] is another interesting one to look at. [0] http://labs.echonest.com/Uploader/index.html http://labs.echonest.com/Uploader/index.html
- tgb 11y agoWow, that is quite good at what it does.
- tgb 11y agoWow, that is quite good at what it does.
- yellowapple 11y agoI love the potentially-infinite loops that are hit on "Rock the Casbah". "ROCK THE CASBAH ROCK THE CASBAH ROCK THE CASBAH ROCK THE CASBAH ROCK THE CASBAH".