4 ms·

Specifically, for these kinds of sequences (chord sequences, etc.) check out models that uses note-level features but with joint probability (RNN with sigmoid o

Specifically, for these kinds of sequences (chord sequences, etc.) check out models that uses note-level features but with joint probability (RNN with sigmoid out and cross-entropy loss, RBM, NADE, etc.). It should do even better!

thanatropism 11y ago
So I took a class in college (15 years ago) where we learned how to do some backprop (out of plain Matlab). I've been seeing impressive demonstrations of RNNs these weeks. How (if at all) can a Matlab-Fortran-SAS grunt start-from-the-beginning to at least understand the rudiments of how these work?
- kastnerkyle 11y ago
  There is a nice book by my profs (Y. Bengio, A. Courville) and a former student of the lab (I. Goodfellow) here: http://www.iro.umontreal.ca/~bengioy/DLbook/ http://www.iro.umontreal.ca/~bengioy/DLbook/ The chapter on RNNs is pretty enlightening. You probably want to start with something the goes through feedforward networks, convolutional, etc. first though. Andrew Ng's Coursera course does a super basic NN in MATLAB which might be good to shake the rust off [1]. Hugo Larochelle's youtube course [2], Geoff Hinton's Coursera course [3], and Nando's youtube Deep Learning course [4] are all very good. There are also some great lectures from Aaron Courville from the Representation Learning class that just finished here at UdeM [5] If you want to do this for real (not self teaching RNN backprop on toy examples) tool-wise you generally go with either Theano (my preference) or Torch - implementing the backwards path for these (and verifying gradients are correct) is not pretty. Theano does it almost for free, and Torch can do it at the module level. Either way you definitely want one of the higher order tools! [1] https://www.coursera.org/learn/machine-learning https://www.coursera.org/learn/machine-learning [2] https://www.youtube.com/watch?v=SGZ6BttHMPw&list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH https://www.youtube.com/watch?v=SGZ6BttHMPw&list=PL6Xpj9I5qX... [3] https://www.coursera.org/course/neuralnets https://www.coursera.org/course/neuralnets [4] https://www.youtube.com/watch?v=PlhFWT7vAEw https://www.youtube.com/watch?v=PlhFWT7vAEw [5] https://ift6266h15.wordpress.com/ https://ift6266h15.wordpress.com/
  - thanatropism 11y ago
    Thanks for the references!

thanatropism 11y ago

So I took a class in college (15 years ago) where we learned how to do some backprop (out of plain Matlab). I've been seeing impressive demonstrations of RNNs these weeks. How (if at all) can a Matlab-Fortran-SAS grunt start-from-the-beginning to at least understand the rudiments of how these work?

kastnerkyle 11y ago
There is a nice book by my profs (Y. Bengio, A. Courville) and a former student of the lab (I. Goodfellow) here: http://www.iro.umontreal.ca/~bengioy/DLbook/ http://www.iro.umontreal.ca/~bengioy/DLbook/ The chapter on RNNs is pretty enlightening. You probably want to start with something the goes through feedforward networks, convolutional, etc. first though. Andrew Ng's Coursera course does a super basic NN in MATLAB which might be good to shake the rust off [1]. Hugo Larochelle's youtube course [2], Geoff Hinton's Coursera course [3], and Nando's youtube Deep Learning course [4] are all very good. There are also some great lectures from Aaron Courville from the Representation Learning class that just finished here at UdeM [5] If you want to do this for real (not self teaching RNN backprop on toy examples) tool-wise you generally go with either Theano (my preference) or Torch - implementing the backwards path for these (and verifying gradients are correct) is not pretty. Theano does it almost for free, and Torch can do it at the module level. Either way you definitely want one of the higher order tools! [1] https://www.coursera.org/learn/machine-learning https://www.coursera.org/learn/machine-learning [2] https://www.youtube.com/watch?v=SGZ6BttHMPw&list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH https://www.youtube.com/watch?v=SGZ6BttHMPw&list=PL6Xpj9I5qX... [3] https://www.coursera.org/course/neuralnets https://www.coursera.org/course/neuralnets [4] https://www.youtube.com/watch?v=PlhFWT7vAEw https://www.youtube.com/watch?v=PlhFWT7vAEw [5] https://ift6266h15.wordpress.com/ https://ift6266h15.wordpress.com/
- thanatropism 11y ago
  Thanks for the references!