3 ms·

Because it's much easier to work with and improve something if there's a theory behind it? There's a spectrum between "just keep trying tons of crap and see wh

Because it's much easier to work with and improve something if there's a theory behind it? There's a spectrum between "just keep trying tons of crap and see what works" and "do this simple, well-understood calculation to see exactly what will work." Is it not obvious why it's nicer to be on the latter end of the spectrum than the former?

bitL 9y ago
Sure, but currently deep learning is more like experimental physics. You try stuff and see what works and empirically improve your understanding. Then you can generalize some heuristics from this and use that "recipe" in the future. You figured out ReLU suddenly made something work, yet Swish turned out better so you can forget about ReLU now. And as you can treat deep learning (in supervised mode) as non-linear optimization, I doubt we'll come up with a proper theory unless P=NP. We can't even understand far simpler non-linear optimization problems, not to mention ones which can be arbitrarily parametrized in 2nd order...
- mturmon 9y ago
  But to just restate the comment you're replying to, clearly theory can prune a lot of branches on the "iterative experiment-based design refinement" method you are proposing. Also, I'll mention that you're being too pessimistic about what theory can accomplish. The learning problem is much more constrained than P = NP. The use, for example, of a training set of N examples, drawn iid, and evaluation on samples drawn from the same distribution imposes a lot of structure. I don't know if you're familiar with VC theory, but it's an example of the kind of "surprising" guarantees that can be derived in this setting. Other general examples are weak learning, the bias/variance tradeoff, and (in SVMs) the notion of large margin classifiers. An applicable "theory" of design is what separates engineering from just mucking around.
- enord 9y ago
  Inscrutable models? Check. "I know it when i see it" success criteria? Check. Unjustified optimism? Check. Sounds more like alchemy than experimental physics to me.
- YeGoblynQueenne 9y ago
  >> Sure, but currently deep learning is more like experimental physics. You try stuff and see what works and empirically improve your understanding. The way I understand experimental physics, its purpose is either (a) to make observations on which new theories can be built or (b) to experimentally verify, or falsify, existing theories. I don't think it's quite like throwing stuff at a wall and looking to see what sticks.

bitL 9y ago

Sure, but currently deep learning is more like experimental physics. You try stuff and see what works and empirically improve your understanding. Then you can generalize some heuristics from this and use that "recipe" in the future. You figured out ReLU suddenly made something work, yet Swish turned out better so you can forget about ReLU now. And as you can treat deep learning (in supervised mode) as non-linear optimization, I doubt we'll come up with a proper theory unless P=NP. We can't even understand far simpler non-linear optimization problems, not to mention ones which can be arbitrarily parametrized in 2nd order...

mturmon 9y ago
But to just restate the comment you're replying to, clearly theory can prune a lot of branches on the "iterative experiment-based design refinement" method you are proposing. Also, I'll mention that you're being too pessimistic about what theory can accomplish. The learning problem is much more constrained than P = NP. The use, for example, of a training set of N examples, drawn iid, and evaluation on samples drawn from the same distribution imposes a lot of structure. I don't know if you're familiar with VC theory, but it's an example of the kind of "surprising" guarantees that can be derived in this setting. Other general examples are weak learning, the bias/variance tradeoff, and (in SVMs) the notion of large margin classifiers. An applicable "theory" of design is what separates engineering from just mucking around.
enord 9y ago
Inscrutable models? Check. "I know it when i see it" success criteria? Check. Unjustified optimism? Check. Sounds more like alchemy than experimental physics to me.
YeGoblynQueenne 9y ago
>> Sure, but currently deep learning is more like experimental physics. You try stuff and see what works and empirically improve your understanding. The way I understand experimental physics, its purpose is either (a) to make observations on which new theories can be built or (b) to experimentally verify, or falsify, existing theories. I don't think it's quite like throwing stuff at a wall and looking to see what sticks.