2 ms·

That’s such a fascinating background to have! It must be strange to retire and, far from the cliché of your skills having been made obsolete and irrelevant, ins

by samhw 4y ago

That’s such a fascinating background to have! It must be strange to retire and, far from the cliché of your skills having been made obsolete and irrelevant, instead you’re an expert on what’s probably the forefront of modern technology. If you can spare a moment to answer: is there any knowledge from the 80s neural net ‘summer’ which you think has been forgotten now? As someone who’s concerned by both poor performance and overfitting (strongly correlated), I thought it was a shame that lots of the research around pruning (optimal brain damage, optimal brain surgeon, etc) has been forgotten[0]. [0] It feels to me as though the ML community has convinced itself - in defiance of information theory - that highly overparameterised models are totally OK, and can even successfully extrapolate with greater than random accuracy in the general case (‘double descent’). I worry that lots of these models are effectively succeeding only at interpolation problems, and only by virtue of the massive hardware advances that let us memorise the entire training set in these enormous models - basically Runge’s phenomenon writ large. People seem to be convincing themselves of magical things that are not mathematically sound.

dougabug 4y ago
Pruning ( both structured and unstructured) is an extremely active area of research and development. Since the Ampere generation, Nvidia GPU’s have HW support for sparsified networks. WRT to overparameterized models, even the largest models such as Google’s PALM 540b model has several orders of magnitude fewer connections than the human brain. Remembering a set of observations doesn’t forbid deducing a general pattern. Imagine trying to understand why the Pythagorean Theorem is true without first memorizing what it states! Children often benefit from seeing and even memorizing many concrete examples before being taught the abstract principles which explain them all. Kepler deduced his Three Laws of Planetary Motion by analyzing four decades of careful astronomical observations compiled by Tycho Brahe. Implicit and explicit regularizations bias modern learning algorithms to seek generally simpler solutions in “wide flat basins of the loss surface.” For instance, the humble skip connection acts to help smooth out the loss surface.

by samhw 4y ago

dougabug 4y ago

Pruning ( both structured and unstructured) is an extremely active area of research and development. Since the Ampere generation, Nvidia GPU’s have HW support for sparsified networks. WRT to overparameterized models, even the largest models such as Google’s PALM 540b model has several orders of magnitude fewer connections than the human brain. Remembering a set of observations doesn’t forbid deducing a general pattern. Imagine trying to understand why the Pythagorean Theorem is true without first memorizing what it states! Children often benefit from seeing and even memorizing many concrete examples before being taught the abstract principles which explain them all. Kepler deduced his Three Laws of Planetary Motion by analyzing four decades of careful astronomical observations compiled by Tycho Brahe. Implicit and explicit regularizations bias modern learning algorithms to seek generally simpler solutions in “wide flat basins of the loss surface.” For instance, the humble skip connection acts to help smooth out the loss surface.