5 ms·

So, i have a very simple question.. : I want to start deep learning now, to implement RNNs, autoencoders, Q-learning on a very specific application (not images

So, i have a very simple question.. : I want to start deep learning now, to implement RNNs, autoencoders, Q-learning on a very specific application (not images). I've read a lot of papers, but not done any DL implementations yet (although many classical ML so far), my question is very simple : Where do I start ???? Should I use Torch ? Theano ? Theano + pylearn2 ? Theano + pylearn + Lasagne ? Caffe ? Or should I just switch to this magical new library directly ? I feel confused.. any advice ?

varelse 11y ago
Methinks embrace the magical library written by engineers and researchers at Google instead of by a random goulash of machine learning grad students with no investment in the outcome other than graduating.
- rhaps0dy 11y ago
  > with no investment in the outcome other than graduating Come on. Grad students are motivated by advancing the field. ... are they not? Maybe you can also be saying that the ones who write libraries are motivated only by graduating.
  - varelse 11y ago
    Sure, we all start out that way. But by the 3rd or 4th year, you frequently just want to write that thesis and get on to the next thing because you've already advanced the field as much as you're going to as a grad student. I have a large package of code on sourceforge that consists of my grad school and postdoctoral efforts. If I compare its style to what 2 decades in the industry since has taught me, it's laughable. That said, it is C code, so I occasionally plunder it for algorithms I figured out way back when.
    - benanne 11y ago
      Nobody forced us to open-source Lasagne, so I think that remark was a bit unfair. If we really didn't care about anything but graduating, why would we bother going through the trouble of sharing the code in the first place? But I do see your point. Google obviously has a lot more manpower to spend on this, so it might be a better bet in the long run. It's also worth comparing this to a few similar projects that have been announced recently: MXNet (http://mxnet.readthedocs.org/en/latest/ http://mxnet.readthedocs.org/en/latest/), Chainer (http://chainer.org/ http://chainer.org/) and CGT (http://rll.berkeley.edu/cgt/ http://rll.berkeley.edu/cgt/). And Theano of course, which has been the flag-bearer of the computational graph approach in deep learning for many years.
      - varelse 11y ago
        Actually lasagne is pretty good, I wasn't targeting you (but then you're pretty good at winning kaggle competitions so perhaps there's a connection there, no?)... I'm thinking mostly of Theano, which, from a performance standpoint, appears to have died the death of a thousand inexperienced cooks in the kitchen. The ~1000x performance regressions that it invokes when a junior data scientist goes off the rails and ends up with a python inner loop amidst GPU kernels is just depressing and seemingly unfixable. Hopefully, TensorFlow will be better if only because it was written in a world now very aware of Pixel's Law. Mxnet is awesome, but perhaps a little too parameter servery for my personal tastes, and I'm now wondering what the point of CGT is now other than to be Coke to Google's Pepsi. I also think the whole deep learning framework business model just took a torpedo amidships (and not long after the layoffs at Ersatz). Finally, I had never heard of Chainer until today, thanks! That said, without autograd functionality, the people I work with would probably stick with Caffe + cuDNN.
        robbyetor 11y ago
        Pixel's Law? what s that? can' t find it on google...
        varelse 11y ago
        Interesting, it does appear to be gone... Basically, old NVIDIA marketing material used to state that GPUs double in performance every year or so whilst Moore's Law is slowing down w/r to CPU performance because clock speeds have been the same for a long time This strictly isn't true because core count, SIMD width and vector unit counts in CPUs have all been increasing. However, from the perspective of a single-threaded C application, this is indeed so. CUDA/OpenCL OTOH automagically subsume multi-core, multi-threading, and SIMD into the language itself so the hypothetical "single-threaded" CUDA app just keeps getting better(tm). The reality though (IMO of course) is that Intel promises and delivers backwards-compatibility at the expense of free performance beer. In contrast, NVIDIA delivers performance at the expense of 100% backwards-compatibility beer for optimized code (but read each GPU's whitepaper and spend a week refactoring your code per GPU generation and you get both, also IMO and of course experience). Of course, to be fair, if you refactor your code every time they improve AVX/SSE, CPUs are a lot mightier than what Python/Javascript/R usually imply.
- pilooch 11y ago
  Well, wait to see how support goes etc... In the meantime take on Keras or Lasagne, and do your stuff, i.e. acquire hands-on experience now. When time comes that you need to go into production, or you need a model that is not available in your current lib or framework, then consider a switch. Point being that experience acquired with one framework will empower you whatsover. In my view, these frameworks are like programming languages, you do need to pick a new one up from time to time, but you can do most of the things with all of them.
pvnick 11y ago
Right now Keras is your best bet for most deep learning tasks, and Caffe is best for image processing (i.e. convolutional neural networks). Keep an eye on tensorflow though because it may come to be the gold standard over the next several months to year. Edit: Btw, François Chollet, the author of keras, made the following tweet this morning: "For those wondering about Keras and TensorFlow: I hope to start working on porting Keras to TensorFlow soon (Theano support will continue)."

varelse 11y ago

Methinks embrace the magical library written by engineers and researchers at Google instead of by a random goulash of machine learning grad students with no investment in the outcome other than graduating.

rhaps0dy 11y ago
> with no investment in the outcome other than graduating Come on. Grad students are motivated by advancing the field. ... are they not? Maybe you can also be saying that the ones who write libraries are motivated only by graduating.
- varelse 11y ago
  Sure, we all start out that way. But by the 3rd or 4th year, you frequently just want to write that thesis and get on to the next thing because you've already advanced the field as much as you're going to as a grad student. I have a large package of code on sourceforge that consists of my grad school and postdoctoral efforts. If I compare its style to what 2 decades in the industry since has taught me, it's laughable. That said, it is C code, so I occasionally plunder it for algorithms I figured out way back when.
  - benanne 11y ago
    Nobody forced us to open-source Lasagne, so I think that remark was a bit unfair. If we really didn't care about anything but graduating, why would we bother going through the trouble of sharing the code in the first place? But I do see your point. Google obviously has a lot more manpower to spend on this, so it might be a better bet in the long run. It's also worth comparing this to a few similar projects that have been announced recently: MXNet (http://mxnet.readthedocs.org/en/latest/ http://mxnet.readthedocs.org/en/latest/), Chainer (http://chainer.org/ http://chainer.org/) and CGT (http://rll.berkeley.edu/cgt/ http://rll.berkeley.edu/cgt/). And Theano of course, which has been the flag-bearer of the computational graph approach in deep learning for many years.
    - varelse 11y ago
      Actually lasagne is pretty good, I wasn't targeting you (but then you're pretty good at winning kaggle competitions so perhaps there's a connection there, no?)... I'm thinking mostly of Theano, which, from a performance standpoint, appears to have died the death of a thousand inexperienced cooks in the kitchen. The ~1000x performance regressions that it invokes when a junior data scientist goes off the rails and ends up with a python inner loop amidst GPU kernels is just depressing and seemingly unfixable. Hopefully, TensorFlow will be better if only because it was written in a world now very aware of Pixel's Law. Mxnet is awesome, but perhaps a little too parameter servery for my personal tastes, and I'm now wondering what the point of CGT is now other than to be Coke to Google's Pepsi. I also think the whole deep learning framework business model just took a torpedo amidships (and not long after the layoffs at Ersatz). Finally, I had never heard of Chainer until today, thanks! That said, without autograd functionality, the people I work with would probably stick with Caffe + cuDNN.
      - robbyetor 11y ago
        Pixel's Law? what s that? can' t find it on google...
        varelse 11y ago
        Interesting, it does appear to be gone... Basically, old NVIDIA marketing material used to state that GPUs double in performance every year or so whilst Moore's Law is slowing down w/r to CPU performance because clock speeds have been the same for a long time This strictly isn't true because core count, SIMD width and vector unit counts in CPUs have all been increasing. However, from the perspective of a single-threaded C application, this is indeed so. CUDA/OpenCL OTOH automagically subsume multi-core, multi-threading, and SIMD into the language itself so the hypothetical "single-threaded" CUDA app just keeps getting better(tm). The reality though (IMO of course) is that Intel promises and delivers backwards-compatibility at the expense of free performance beer. In contrast, NVIDIA delivers performance at the expense of 100% backwards-compatibility beer for optimized code (but read each GPU's whitepaper and spend a week refactoring your code per GPU generation and you get both, also IMO and of course experience). Of course, to be fair, if you refactor your code every time they improve AVX/SSE, CPUs are a lot mightier than what Python/Javascript/R usually imply.
pilooch 11y ago
Well, wait to see how support goes etc... In the meantime take on Keras or Lasagne, and do your stuff, i.e. acquire hands-on experience now. When time comes that you need to go into production, or you need a model that is not available in your current lib or framework, then consider a switch. Point being that experience acquired with one framework will empower you whatsover. In my view, these frameworks are like programming languages, you do need to pick a new one up from time to time, but you can do most of the things with all of them.

pvnick 11y ago

Right now Keras is your best bet for most deep learning tasks, and Caffe is best for image processing (i.e. convolutional neural networks). Keep an eye on tensorflow though because it may come to be the gold standard over the next several months to year. Edit: Btw, François Chollet, the author of keras, made the following tweet this morning: "For those wondering about Keras and TensorFlow: I hope to start working on porting Keras to TensorFlow soon (Theano support will continue)."