21 ms·

> with no investment in the outcome other than graduating Come on. Grad students are motivated by advancing the field. ... are they not? Maybe you can also be

> with no investment in the outcome other than graduating Come on. Grad students are motivated by advancing the field. ... are they not? Maybe you can also be saying that the ones who write libraries are motivated only by graduating.

varelse 11y ago
Sure, we all start out that way. But by the 3rd or 4th year, you frequently just want to write that thesis and get on to the next thing because you've already advanced the field as much as you're going to as a grad student. I have a large package of code on sourceforge that consists of my grad school and postdoctoral efforts. If I compare its style to what 2 decades in the industry since has taught me, it's laughable. That said, it is C code, so I occasionally plunder it for algorithms I figured out way back when.
- benanne 11y ago
  Nobody forced us to open-source Lasagne, so I think that remark was a bit unfair. If we really didn't care about anything but graduating, why would we bother going through the trouble of sharing the code in the first place? But I do see your point. Google obviously has a lot more manpower to spend on this, so it might be a better bet in the long run. It's also worth comparing this to a few similar projects that have been announced recently: MXNet (http://mxnet.readthedocs.org/en/latest/ http://mxnet.readthedocs.org/en/latest/), Chainer (http://chainer.org/ http://chainer.org/) and CGT (http://rll.berkeley.edu/cgt/ http://rll.berkeley.edu/cgt/). And Theano of course, which has been the flag-bearer of the computational graph approach in deep learning for many years.
  - varelse 11y ago
    Actually lasagne is pretty good, I wasn't targeting you (but then you're pretty good at winning kaggle competitions so perhaps there's a connection there, no?)... I'm thinking mostly of Theano, which, from a performance standpoint, appears to have died the death of a thousand inexperienced cooks in the kitchen. The ~1000x performance regressions that it invokes when a junior data scientist goes off the rails and ends up with a python inner loop amidst GPU kernels is just depressing and seemingly unfixable. Hopefully, TensorFlow will be better if only because it was written in a world now very aware of Pixel's Law. Mxnet is awesome, but perhaps a little too parameter servery for my personal tastes, and I'm now wondering what the point of CGT is now other than to be Coke to Google's Pepsi. I also think the whole deep learning framework business model just took a torpedo amidships (and not long after the layoffs at Ersatz). Finally, I had never heard of Chainer until today, thanks! That said, without autograd functionality, the people I work with would probably stick with Caffe + cuDNN.
    - robbyetor 11y ago
      Pixel's Law? what s that? can' t find it on google...
      - varelse 11y ago
        Interesting, it does appear to be gone... Basically, old NVIDIA marketing material used to state that GPUs double in performance every year or so whilst Moore's Law is slowing down w/r to CPU performance because clock speeds have been the same for a long time This strictly isn't true because core count, SIMD width and vector unit counts in CPUs have all been increasing. However, from the perspective of a single-threaded C application, this is indeed so. CUDA/OpenCL OTOH automagically subsume multi-core, multi-threading, and SIMD into the language itself so the hypothetical "single-threaded" CUDA app just keeps getting better(tm). The reality though (IMO of course) is that Intel promises and delivers backwards-compatibility at the expense of free performance beer. In contrast, NVIDIA delivers performance at the expense of 100% backwards-compatibility beer for optimized code (but read each GPU's whitepaper and spend a week refactoring your code per GPU generation and you get both, also IMO and of course experience). Of course, to be fair, if you refactor your code every time they improve AVX/SSE, CPUs are a lot mightier than what Python/Javascript/R usually imply.

varelse 11y ago

Sure, we all start out that way. But by the 3rd or 4th year, you frequently just want to write that thesis and get on to the next thing because you've already advanced the field as much as you're going to as a grad student. I have a large package of code on sourceforge that consists of my grad school and postdoctoral efforts. If I compare its style to what 2 decades in the industry since has taught me, it's laughable. That said, it is C code, so I occasionally plunder it for algorithms I figured out way back when.

benanne 11y ago
Nobody forced us to open-source Lasagne, so I think that remark was a bit unfair. If we really didn't care about anything but graduating, why would we bother going through the trouble of sharing the code in the first place? But I do see your point. Google obviously has a lot more manpower to spend on this, so it might be a better bet in the long run. It's also worth comparing this to a few similar projects that have been announced recently: MXNet (http://mxnet.readthedocs.org/en/latest/ http://mxnet.readthedocs.org/en/latest/), Chainer (http://chainer.org/ http://chainer.org/) and CGT (http://rll.berkeley.edu/cgt/ http://rll.berkeley.edu/cgt/). And Theano of course, which has been the flag-bearer of the computational graph approach in deep learning for many years.
- varelse 11y ago
  Actually lasagne is pretty good, I wasn't targeting you (but then you're pretty good at winning kaggle competitions so perhaps there's a connection there, no?)... I'm thinking mostly of Theano, which, from a performance standpoint, appears to have died the death of a thousand inexperienced cooks in the kitchen. The ~1000x performance regressions that it invokes when a junior data scientist goes off the rails and ends up with a python inner loop amidst GPU kernels is just depressing and seemingly unfixable. Hopefully, TensorFlow will be better if only because it was written in a world now very aware of Pixel's Law. Mxnet is awesome, but perhaps a little too parameter servery for my personal tastes, and I'm now wondering what the point of CGT is now other than to be Coke to Google's Pepsi. I also think the whole deep learning framework business model just took a torpedo amidships (and not long after the layoffs at Ersatz). Finally, I had never heard of Chainer until today, thanks! That said, without autograd functionality, the people I work with would probably stick with Caffe + cuDNN.
  - robbyetor 11y ago
    Pixel's Law? what s that? can' t find it on google...
    - varelse 11y ago
      Interesting, it does appear to be gone... Basically, old NVIDIA marketing material used to state that GPUs double in performance every year or so whilst Moore's Law is slowing down w/r to CPU performance because clock speeds have been the same for a long time This strictly isn't true because core count, SIMD width and vector unit counts in CPUs have all been increasing. However, from the perspective of a single-threaded C application, this is indeed so. CUDA/OpenCL OTOH automagically subsume multi-core, multi-threading, and SIMD into the language itself so the hypothetical "single-threaded" CUDA app just keeps getting better(tm). The reality though (IMO of course) is that Intel promises and delivers backwards-compatibility at the expense of free performance beer. In contrast, NVIDIA delivers performance at the expense of 100% backwards-compatibility beer for optimized code (but read each GPU's whitepaper and spend a week refactoring your code per GPU generation and you get both, also IMO and of course experience). Of course, to be fair, if you refactor your code every time they improve AVX/SSE, CPUs are a lot mightier than what Python/Javascript/R usually imply.