4 ms·

This is a fantastic answer - thank you, Eli. So what do you think of the original article?

eli_gottlieb 9y ago
>This is a fantastic answer - thank you, Eli. Thanks! I've been doing a lot of amateur reading in cog-sci and theoretical neurosci. The subject enthuses me enough that I'm applying to PhD programs in it this upcoming season. >So what do you think of the original article? Thorough and accurate. I'll give a little expansion via my own thought. One thing taught in every theoretically-focused ML class is the No Free Lunch Theorem. In colloquial terms it says, "If you don't make some simplifying assumptions about the function you're trying to learn (and the distribution noising your data), you can't reliably learn." I think experts learn this, appreciate it as a point of theory, and then often forget to really bring it back up and rethink it where it's applicable. All statistical learning takes place subject to assumptions of "niceness". Which assumptions, though? Seems to me like: * If you make certain "niceness" assumptions about the functions in your hypothesis space, but few to none about the distribution, you're a Machine Learner. * If you make niceness assumptions about your distribution, but don't quite care about the generating function itself, you're an Applied Statistician. * If you make niceness assumptions about your data, that it was generated from some family of distributions on which you can make inferences, you're a fully frequentist or Bayesian statistician. * If you want to make almost no assumptions about the generating process yielding the data, but still want just enough assumptions to make reasoning possible, you may be working in the vicinity of any of cognitive science, neuroscience, or artificial intelligence. The key thing you always have to remind yourself is: you are making assumptions. The question is: which ones? The original article reminds us of a whole lot of the assumptions behind current deep learning: * The "layers" we care about are compositions of a continuous nonlinear function with a linear transform. * The functions we care about are compositions of "layers". * The transforms we care about are probably convolutions or just linear-and-rectified, or just linear-and-sigmoid. * Composing layers enables gradient information to "fan out" from the loss function to wider and wider places in the early layers. * The data spaces we care about are usually Euclidean. These are things every expert knows, but which most people only question when it's time to look at the limitations of current methods. The author of the original article appears well-versed in everything, and I'm really excited to see what they've got for the next part.

>This is a fantastic answer - thank you, Eli. Thanks! I've been doing a lot of amateur reading in cog-sci and theoretical neurosci. The subject enthuses me enough that I'm applying to PhD programs in it this upcoming season. >So what do you think of the original article? Thorough and accurate. I'll give a little expansion via my own thought. One thing taught in every theoretically-focused ML class is the No Free Lunch Theorem. In colloquial terms it says, "If you don't make some simplifying assumptions about the function you're trying to learn (and the distribution noising your data), you can't reliably learn." I think experts learn this, appreciate it as a point of theory, and then often forget to really bring it back up and rethink it where it's applicable. All statistical learning takes place subject to assumptions of "niceness". Which assumptions, though? Seems to me like: * If you make certain "niceness" assumptions about the functions in your hypothesis space, but few to none about the distribution, you're a Machine Learner. * If you make niceness assumptions about your distribution, but don't quite care about the generating function itself, you're an Applied Statistician. * If you make niceness assumptions about your data, that it was generated from some family of distributions on which you can make inferences, you're a fully frequentist or Bayesian statistician. * If you want to make almost no assumptions about the generating process yielding the data, but still want just enough assumptions to make reasoning possible, you may be working in the vicinity of any of cognitive science, neuroscience, or artificial intelligence. The key thing you always have to remind yourself is: you are making assumptions. The question is: which ones? The original article reminds us of a whole lot of the assumptions behind current deep learning: * The "layers" we care about are compositions of a continuous nonlinear function with a linear transform. * The functions we care about are compositions of "layers". * The transforms we care about are probably convolutions or just linear-and-rectified, or just linear-and-sigmoid. * Composing layers enables gradient information to "fan out" from the loss function to wider and wider places in the early layers. * The data spaces we care about are usually Euclidean. These are things every expert knows, but which most people only question when it's time to look at the limitations of current methods. The author of the original article appears well-versed in everything, and I'm really excited to see what they've got for the next part.