3 ms·

A bit OT: I've always wondered - you see a lot of relatively young engineers and devs doing work in what seems to be cutting edge DL work, and some of these are

A bit OT: I've always wondered - you see a lot of relatively young engineers and devs doing work in what seems to be cutting edge DL work, and some of these are even self-thought, with sometimes only a few years of experience (at least if their open github accounts and blog resumes are any good gauge of experience). On the other hand - the only (really) talented people I personally know, that are working on Deep Learning problems, all have heavy and very solid academic backgrounds from Math / Physics / CS / etc., and are either finishing up their Ph.D's, or were done the past few years. Most started studying (or working on) ML before the Deep Learning craze took off. All in all, these guys have at least a decade of solid experience - but in return, know a metric ton of classic ML / stats / grad. level math / etc. by heart. Obviously, in probably most cases, one can't compress 10 years of hard work experience into 1-2 years, even though that's what a lot of these "roadmaps" seem to promise. Makes me wonder if the internet-world of deep learning is filled with savants that are able to catch up with decades of research and best-practices in only couple of years, or if these are people that are really, really specialized in some niche application of DL.

klipklop 6y ago
Seems to me humans are really good at 'catching up' based on what we perceive as possible. For example when a new world record is set in a sport often the idea of such a performance is possible raises the bar for the entire field rapidly. Within months multiple competitors are nipping at the heels of the record. I also see this in video game speed running. When a new breakthrough record is set suddenly the competitors begin to catch up to the new record now that they see it is indeed possible. Even new people playing the game don't take 10 years to catch up skill wise when they can learn from previous attempts from the 'old masters.' It seems like even in the arena of AI breakthroughs as soon as some paper gets published others quickly catch up. I do agree there is no replacement for 10 years of research and work, but future people studying in the field can and do catch up.
scsilver 6y ago
Trailblazers will always make slower progress than those following in their footsteps. It took researchers how long to develop calculus and we become effective at wielding it in 3 classes
cycrutchfield 6y ago
There's a huge difference between being able to apply a set of known techniques and being able to create novel techniques. I suspect most of the autodidacts you are referring to are able to read papers and implement them, and are well-versed in the practical aspects of how to tune models, but would likely struggle with doing something novel enough that is actually worth publishing. There are exceptions, of course, but for certain types of work that extensive background in math/stats/cs is absolutely necessary.
saeranv 6y ago
I feel like it's more complementary then that. You need the PhDs to develop core theories and technology, and you need the young engineers and devs to understand it enough to apply it in other domains. Both can be described as doing "cutting-edge DL work" but just in different domains. And, the young devs don't need to try and catch up, they need to be generalists.
ilaksh 6y ago
It seems cutting-edge how? Because for example you can literally go to GitHub and get a project with a complex deep learning architecture close to the state of the art (or at least last year's state of the art) for some things. Then pull that up on your screen and start messing with the structure of the graph, adding layers etc. with impressive sounding names like "relu" etc. Lol. It could look cutting edge and it actually could kind of be cutting edge as far as the fact that regardless of your background the next step to making an incremental improvement in an area may be taking a known model and tweaking the hyperparameters and assembling a good dataset. But obviously there is a difference between that and creating a new architecture or approach from scratch. In other words, you are not going to invent GANs or capsules by tweaking hyperparameters. Or come up with other more basic innovations necessarily without a deep understanding of the underlying concepts. However I will say that the repeated practical application of deep learning seems like it is going to be a critical part of a having a strong grasp. And someone with a Ph.D in ML mifht actually be less useful for a particular application than a person who has actually used Keras for ten different real projects. At the core, many people are involved in applying existing approaches by assembling datasets and tweaking hyperparameters and architectures. Which maybe amounts to modifying a few hundred or maybe dozen lines of Python code that describe the different layers and learning rates etc. and load the data. People can leverage programming libraries and be more effective than another guy who has less experience with that library but is a true expert in statistics and LaTeX.

klipklop 6y ago

Seems to me humans are really good at 'catching up' based on what we perceive as possible. For example when a new world record is set in a sport often the idea of such a performance is possible raises the bar for the entire field rapidly. Within months multiple competitors are nipping at the heels of the record. I also see this in video game speed running. When a new breakthrough record is set suddenly the competitors begin to catch up to the new record now that they see it is indeed possible. Even new people playing the game don't take 10 years to catch up skill wise when they can learn from previous attempts from the 'old masters.' It seems like even in the arena of AI breakthroughs as soon as some paper gets published others quickly catch up. I do agree there is no replacement for 10 years of research and work, but future people studying in the field can and do catch up.

scsilver 6y ago

Trailblazers will always make slower progress than those following in their footsteps. It took researchers how long to develop calculus and we become effective at wielding it in 3 classes

cycrutchfield 6y ago

There's a huge difference between being able to apply a set of known techniques and being able to create novel techniques. I suspect most of the autodidacts you are referring to are able to read papers and implement them, and are well-versed in the practical aspects of how to tune models, but would likely struggle with doing something novel enough that is actually worth publishing. There are exceptions, of course, but for certain types of work that extensive background in math/stats/cs is absolutely necessary.

saeranv 6y ago

I feel like it's more complementary then that. You need the PhDs to develop core theories and technology, and you need the young engineers and devs to understand it enough to apply it in other domains. Both can be described as doing "cutting-edge DL work" but just in different domains. And, the young devs don't need to try and catch up, they need to be generalists.

ilaksh 6y ago

It seems cutting-edge how? Because for example you can literally go to GitHub and get a project with a complex deep learning architecture close to the state of the art (or at least last year's state of the art) for some things. Then pull that up on your screen and start messing with the structure of the graph, adding layers etc. with impressive sounding names like "relu" etc. Lol. It could look cutting edge and it actually could kind of be cutting edge as far as the fact that regardless of your background the next step to making an incremental improvement in an area may be taking a known model and tweaking the hyperparameters and assembling a good dataset. But obviously there is a difference between that and creating a new architecture or approach from scratch. In other words, you are not going to invent GANs or capsules by tweaking hyperparameters. Or come up with other more basic innovations necessarily without a deep understanding of the underlying concepts. However I will say that the repeated practical application of deep learning seems like it is going to be a critical part of a having a strong grasp. And someone with a Ph.D in ML mifht actually be less useful for a particular application than a person who has actually used Keras for ten different real projects. At the core, many people are involved in applying existing approaches by assembling datasets and tweaking hyperparameters and architectures. Which maybe amounts to modifying a few hundred or maybe dozen lines of Python code that describe the different layers and learning rates etc. and load the data. People can leverage programming libraries and be more effective than another guy who has less experience with that library but is a true expert in statistics and LaTeX.