4 ms·
My takeaway is something I've observed and commented about for some time: understanding how any important/relevant technology works gives you an unbeatable edge
by varelse 11y ago
My takeaway is something I've observed and commented about for some time: understanding how any important/relevant technology works gives you an unbeatable edge over most of your peers. Andrew Ng makes a similar comment about 2/3 of the way through his Coursera course on machine learning.
With respect to deep learning, most of the work I see involves pulling existing networks off of the Caffe Zoo and retraining them with proprietary datasets or fiddling with Theano to reproduce the works of others. These efforts mostly end in failure, usually after a blind hyperparameter search.
The easiest way to separate the winners from the losers in deep learning and elsewhere is to ask them to derive the method they are using and explain what it's doing. For the most part, I find the above people are incapable of doing so for backpropagation. Bonus points if they can write (relatively) efficient C++/CUDA code to implement backpropagation from the ground up.
Edit the above paragraph for PCA, ICA, GP, GMM, random forests or any other technology/algorithm. If one has a decent grasp of the underlying logic, the instincts for adapting it to new domains/datasets come at no additional charge.
- selectron 11y agoCan you give some anecdotes of where "understanding how any important/relevant technology works gives you an unbeatable edge over most of your peers"? Besides being able to pass a job interview. Naively I would assume that being able to apply these techniques (understanding cross validation, overfitting, domain expertise, etc) would be far more important than knowing the underlying logic of how to write them from scratch.
- johnmaguire2013 11y agoI think that it's way easier to apply something when you understand how that something actually works. i.e. There is a difference between being able to re-implement an algorithm, and understanding why that algorithm has the behavior it has. You'll probably write cleaner and more maintainable code, quicker, in the latter, and be able to make modifications as necessary.
- bearzoo 11y agoFor work a month ago I implemented stochastic gradient descent in some arbitrary directed graph of operations (I applied a skill I learned from neural networks in a system that had nothing to do with neural networks) which required me to know how to take derivatives of a loss function with respect to parameters. I did all of the math on paper, and then coded it and it worked (to be frank this surprised me). It often may be hard to see the immediate gains from understanding things in a more technical, low level fashion - but I think in the long run they give you a great deal of power.