3 ms·

I've said it before and I'll say it again. Machine learning is specifically not magic. It only works to the extent that we can build our own priors into the mod

I've said it before and I'll say it again. Machine learning is specifically not magic. It only works to the extent that we can build our own priors into the model. A typical media story... deep learning really is great. It represents the first time we've really figured out how to do large-scale nonlinear regression. But it is certainly not a magic bullet. However moderate headlines don't get as many hits as overhyped ones so every day we get another ridiculous article spouting nonsense... Very tiresome. The truth is pretty interesting, can we talk about that instead? For instance, how and why deep learning works at ALL is very much an open question. Consider - we're taking an incredibly nonlinear, nonconvex optimization problem and optimizing in just about the dumbest way imaginable, first-order gradient descent. It is really amazing that this works as well as it does. ... Why does deeper work better than wider? It has been known for many years that a shallow net has equivalent expressivity to a deep one. So what gives? (actually some interesting work towards answering this question in recent years by Sohl-Dickenstein et. al)

ggm 9y ago
Here here. Training an algorithm to mimic prior knowledge. The prior knowledge is inherently there. It's like the inflationary claims for QC which depend on demonstrated factorisation to a known product of primes: it's not magic when you seek to minimise error to a known goal but you have to know the goal.
pps43 9y ago
> Machine learning [..] only works to the extent that we can build our own priors into the model. AlphaZero is a good counterexample. In contrast to AlphaGo, it has no priors.
- aoeusnth1 9y ago
  Well it does have some priors baked into it by the (convolutional) architecture of the network. See the “deep image prior” project for a feel for just how strong this convolutional prior is.

ggm 9y ago

Here here. Training an algorithm to mimic prior knowledge. The prior knowledge is inherently there. It's like the inflationary claims for QC which depend on demonstrated factorisation to a known product of primes: it's not magic when you seek to minimise error to a known goal but you have to know the goal.

pps43 9y ago

> Machine learning [..] only works to the extent that we can build our own priors into the model. AlphaZero is a good counterexample. In contrast to AlphaGo, it has no priors.

aoeusnth1 9y ago
Well it does have some priors baked into it by the (convolutional) architecture of the network. See the “deep image prior” project for a feel for just how strong this convolutional prior is.