6 ms·

Interesting to see that the Google Brain Residency isn't that much more advanced than other graduate work or fellowship programs out there. I disagree with him

Interesting to see that the Google Brain Residency isn't that much more advanced than other graduate work or fellowship programs out there. I disagree with him about needing a "Rails for deep learning," I think it's quite fine now in practice, especially since DL code is typically significantly less than a web app. A file for data pipeline, a file for your model, and a file for training and/or inference. Not sure it really needs to be much more complicated than that.

aub3bhat 9y ago
I actually strongly agree with him, In fact the progress in Deep Learning has stalled due to inefficiencies in sharing data and code between researchers. Today to replicate a study you have to spend several days downloading data, making sure all dependencies are installed etc. This is worsened by lack of standard formats for even simple things such as representing 2D bounding box on an image. People forget that the reason Deep Learning came to prominence was engineering that enabled use of GPU's (Thanks to Alex K.) and large dataset from ImageNet competition (Thanks to Fei-Fei Li). Importance of software engineering in moving the field forward is often under-appreciated. This blogpost beautifully illustrates several instances where a great implementation made it easier for researcher to speed up experimentation and lead to breakthroughs in Computer Vision. [1] I am building Deep Video Analytics which aims to become the Rail/Django/MySQL/(favorite analogy) of Visual data Analytics [2,3]. [1] http://www.computervisionblog.com/2015/01/from-feature-descriptors-to-deep.html http://www.computervisionblog.com/2015/01/from-feature-descr... [2] https://github.com/AKSHAYUBHAT/DeepVideoAnalytics/ https://github.com/AKSHAYUBHAT/DeepVideoAnalytics/ [3] http://www.deepvideoanalytics.com/ http://www.deepvideoanalytics.com/
- orthoganol 9y ago
  Well, my professional experience has been completely different, maybe hardcore research in images is different. At any rate isn't TF already the Rails/ magical framework of DL, in some sense?
  - ericd 9y ago
    I'm not an expert, but Keras seems much closer than TF.
    - espadrine 9y ago
      They're complementary. Keras typically uses TF as a back-end, letting users model standard architectures quickly, while TF lets you express new and complex graphs of computation.
      - ericd 9y ago
        I'm aware, I'm using Theano as my backend. I just mean that Keras seems more analogous to Rails than TF - TF would be Ruby in that analogy.
  - shadowmint 9y ago
    I don't know... There's plenty of horrible code out there, particularly tensorflow code, which matches all the classical metrics for 'spaghetti code'. The complexity of the implementation might be hidden away by the framework, but there's no excuse for writing massive hundred+ line functions that do multiple things with copy-pasted code blocks. That's just bad code, in ML or not.
- killjoywashere 9y ago
  I also strongly agree with him, and you. The work of annotating, curating, prepping, etc, is huge. Every industry could split off 10% of its workforce just to work on prepping their data for ingestion, and then designing the interfaces to actually use the inferences the algorithms emit.
ma2rten 9y ago
Interesting to see that the Google Brain Residency isn't that much more advanced than other graduate work or fellowship programs out there. I think they accepted residents from different backgrounds and different levels of experience.
ForHackernews 9y ago
> Interesting to see that the Google Brain Residency isn't that much more advanced than other graduate work or fellowship programs out there. Why would you assume that a temp job at an advertising company would be "more advanced" than work conducted at an actual research institution?
- TheArcane 9y ago
  Even though that's how it makes a lion's share of its revenue, I don't think anyone actually considers Google just an advertising company.
  - ForHackernews 9y ago
    Maybe they don't "consider" Google an advertising company, but that's what it is. Comcast isn't just a cable company these days, but it is still a cable company.
- johnsmith21006 9y ago
  That advertising company did create software that beat the best Go player in the world a decade earlier than estimates. They also created the most power NN processing silicon that uses far less power than anyone else. A pod is doing 11.5 quadrillion FLOPs is pretty impressive. Just sayin.
  - ForHackernews 9y ago
    Sure, stipulated. Google has demonstrated conclusively that if you throw enormous amounts of money and compute infrastructure at problems, you can make progress. Those are still engineering developments, not the kind of groundbreaking research that happens at universities or national labs. It's not even the kind of fundamental research that places like Xerox PARC, IBM, or Bell Labs used to invest in. Frankly, it's kind of insulting to imply that a one-year Google residency would be "more advanced" than years-long graduate programs at places like MIT, Caltech or CMU.

aub3bhat 9y ago

I actually strongly agree with him, In fact the progress in Deep Learning has stalled due to inefficiencies in sharing data and code between researchers. Today to replicate a study you have to spend several days downloading data, making sure all dependencies are installed etc. This is worsened by lack of standard formats for even simple things such as representing 2D bounding box on an image. People forget that the reason Deep Learning came to prominence was engineering that enabled use of GPU's (Thanks to Alex K.) and large dataset from ImageNet competition (Thanks to Fei-Fei Li). Importance of software engineering in moving the field forward is often under-appreciated. This blogpost beautifully illustrates several instances where a great implementation made it easier for researcher to speed up experimentation and lead to breakthroughs in Computer Vision. [1] I am building Deep Video Analytics which aims to become the Rail/Django/MySQL/(favorite analogy) of Visual data Analytics [2,3]. [1] http://www.computervisionblog.com/2015/01/from-feature-descriptors-to-deep.html http://www.computervisionblog.com/2015/01/from-feature-descr... [2] https://github.com/AKSHAYUBHAT/DeepVideoAnalytics/ https://github.com/AKSHAYUBHAT/DeepVideoAnalytics/ [3] http://www.deepvideoanalytics.com/ http://www.deepvideoanalytics.com/

orthoganol 9y ago
Well, my professional experience has been completely different, maybe hardcore research in images is different. At any rate isn't TF already the Rails/ magical framework of DL, in some sense?
- ericd 9y ago
  I'm not an expert, but Keras seems much closer than TF.
  - espadrine 9y ago
    They're complementary. Keras typically uses TF as a back-end, letting users model standard architectures quickly, while TF lets you express new and complex graphs of computation.
    - ericd 9y ago
      I'm aware, I'm using Theano as my backend. I just mean that Keras seems more analogous to Rails than TF - TF would be Ruby in that analogy.
- shadowmint 9y ago
  I don't know... There's plenty of horrible code out there, particularly tensorflow code, which matches all the classical metrics for 'spaghetti code'. The complexity of the implementation might be hidden away by the framework, but there's no excuse for writing massive hundred+ line functions that do multiple things with copy-pasted code blocks. That's just bad code, in ML or not.
killjoywashere 9y ago
I also strongly agree with him, and you. The work of annotating, curating, prepping, etc, is huge. Every industry could split off 10% of its workforce just to work on prepping their data for ingestion, and then designing the interfaces to actually use the inferences the algorithms emit.

ma2rten 9y ago

Interesting to see that the Google Brain Residency isn't that much more advanced than other graduate work or fellowship programs out there. I think they accepted residents from different backgrounds and different levels of experience.

ForHackernews 9y ago

> Interesting to see that the Google Brain Residency isn't that much more advanced than other graduate work or fellowship programs out there. Why would you assume that a temp job at an advertising company would be "more advanced" than work conducted at an actual research institution?

TheArcane 9y ago
Even though that's how it makes a lion's share of its revenue, I don't think anyone actually considers Google just an advertising company.
- ForHackernews 9y ago
  Maybe they don't "consider" Google an advertising company, but that's what it is. Comcast isn't just a cable company these days, but it is still a cable company.
johnsmith21006 9y ago
That advertising company did create software that beat the best Go player in the world a decade earlier than estimates. They also created the most power NN processing silicon that uses far less power than anyone else. A pod is doing 11.5 quadrillion FLOPs is pretty impressive. Just sayin.
- ForHackernews 9y ago
  Sure, stipulated. Google has demonstrated conclusively that if you throw enormous amounts of money and compute infrastructure at problems, you can make progress. Those are still engineering developments, not the kind of groundbreaking research that happens at universities or national labs. It's not even the kind of fundamental research that places like Xerox PARC, IBM, or Bell Labs used to invest in. Frankly, it's kind of insulting to imply that a one-year Google residency would be "more advanced" than years-long graduate programs at places like MIT, Caltech or CMU.