3 ms·

As one of the recent newcomers, should I feel defensive when I read something like this? I understand there are people with much more knowledge. Isn't this tr

As one of the recent newcomers, should I feel defensive when I read something like this? I understand there are people with much more knowledge. Isn't this true of everyone, in every field? The message I've gotten is "try things out". Innovation isn't necessarily improving specific techniques, but applying them to new fields. To apply techniques to things that are more mundane like data processing in non-AI-focused companies, you're gonna need bodies who know how to apply these newer programming techniques to solve problems. Not every electrician has to understand electrical engineering.

nlowell 9y ago
I am new as well. I think the folklore aspect is not because the established people in the field are bad teachers, but because even they don't have much rationale besides "this is what seems to work". It's a new field, that's fine. Innovations are still as simple as "Oh we used a cyclical learning rate instead of constant" and boom.
currymj 9y ago
i don't think so. the author here also helped co-write the "deep learning is alchemy" talk that was somewhat controversial at NIPS. i think this is especially important if you purely want to do applications. we have a bag of tricks (dropout, batchnorm, different optimizers and learning rate schedules). we have no real theory for why any of this should work; often a proposed explanation will later turn out not to make sense. so the choice of how to train things comes down to "folklore", the community's collective experience. and there's no guarantee that folklore will generalize to your new architecture or dataset, and no way to know whether it even should. the presentation seems to have struck a nerve and there's papers and talks floating around now examining the performance of common architectures in very simple settings. it's probably worth paying attention to these at least in the background, as it will hopefully crystallize into a body of knowledge that will be useful for someone trying to decide on architectures and optimization techniques.

nlowell 9y ago

I am new as well. I think the folklore aspect is not because the established people in the field are bad teachers, but because even they don't have much rationale besides "this is what seems to work". It's a new field, that's fine. Innovations are still as simple as "Oh we used a cyclical learning rate instead of constant" and boom.

currymj 9y ago

i don't think so. the author here also helped co-write the "deep learning is alchemy" talk that was somewhat controversial at NIPS. i think this is especially important if you purely want to do applications. we have a bag of tricks (dropout, batchnorm, different optimizers and learning rate schedules). we have no real theory for why any of this should work; often a proposed explanation will later turn out not to make sense. so the choice of how to train things comes down to "folklore", the community's collective experience. and there's no guarantee that folklore will generalize to your new architecture or dataset, and no way to know whether it even should. the presentation seems to have struck a nerve and there's papers and talks floating around now examining the performance of common architectures in very simple settings. it's probably worth paying attention to these at least in the background, as it will hopefully crystallize into a body of knowledge that will be useful for someone trying to decide on architectures and optimization techniques.