6 ms·

I feel like asking: did they solve the problem? Let me see if I can state the problem: Neural Networks are non-linear because of their activation functions. Yo

I feel like asking: did they solve the problem? Let me see if I can state the problem: Neural Networks are non-linear because of their activation functions. You need a differentiable function in order to take the derivative so you can back-prop the error, more or less. The consequence of the non-linearity is that you can't do some kind of short hand calculation to figure out what the network will do. You have to crank through the network to see the result. There is no economy of thought. That is to say, there is no theory. I am excited that they are working on it, but I would love to have a summary or overview of how they approach what I consider to be the basic problem.

alexnewman 9y ago
Is that the only problem? Also why make a network of a certain length. Why not farther. Depth and it’s relation to the degree of the estimated polynomial is well undertood. What’s less understood is why not so deep
- inputcoffee 9y ago
  I didn't mean to say that it is the only problem. It is more like the "meta problem" as far as theory is concerned. There are many parameters (layers, nodes, connectivity, drop off, number of trials etc). Right now they are guessed at by trial and error. The thing preventing you from saying, "Oh, just reduce the layers by 1, and add more nodes" is the non-linearity.
bitL 9y ago
Why is the lack of theory a problem? At some point we have to accept some problems are "out of our league" and use whatever is available even without fully understanding it. We can't even understand simple specializations in computer vision, yet we expect to understand a more general method? It's not like 15 years ago there weren't math theorems proven by enumerating them on a computer. I understand it breaks psyche and pride of some scientists, so what? Universe can't be expected to fit into humanity's collective brain.
- mikebenfield 9y ago
  Because it's much easier to work with and improve something if there's a theory behind it? There's a spectrum between "just keep trying tons of crap and see what works" and "do this simple, well-understood calculation to see exactly what will work." Is it not obvious why it's nicer to be on the latter end of the spectrum than the former?
  - bitL 9y ago
    Sure, but currently deep learning is more like experimental physics. You try stuff and see what works and empirically improve your understanding. Then you can generalize some heuristics from this and use that "recipe" in the future. You figured out ReLU suddenly made something work, yet Swish turned out better so you can forget about ReLU now. And as you can treat deep learning (in supervised mode) as non-linear optimization, I doubt we'll come up with a proper theory unless P=NP. We can't even understand far simpler non-linear optimization problems, not to mention ones which can be arbitrarily parametrized in 2nd order...
    - mturmon 9y ago
      But to just restate the comment you're replying to, clearly theory can prune a lot of branches on the "iterative experiment-based design refinement" method you are proposing. Also, I'll mention that you're being too pessimistic about what theory can accomplish. The learning problem is much more constrained than P = NP. The use, for example, of a training set of N examples, drawn iid, and evaluation on samples drawn from the same distribution imposes a lot of structure. I don't know if you're familiar with VC theory, but it's an example of the kind of "surprising" guarantees that can be derived in this setting. Other general examples are weak learning, the bias/variance tradeoff, and (in SVMs) the notion of large margin classifiers. An applicable "theory" of design is what separates engineering from just mucking around.
    - enord 9y ago
      Inscrutable models? Check. "I know it when i see it" success criteria? Check. Unjustified optimism? Check. Sounds more like alchemy than experimental physics to me.
mikebenfield 9y ago
> Neural Networks are non-linear because of their activation functions. You need a differentiable function in order to take the derivative so you can back-prop the error, more or less. To clarify: linear functions would still be differentiable without an activation function. The problem is that the composition of linear functions is just another linear function, so you gain nothing by having multiple layers; you might as well just do some kind of linear regression or classification. Activation functions introduce nonlinearity so that deep learning methods can hopefully learn things linear methods can't.
- inputcoffee 9y ago
  When I wrote that, I was thinking of the unit step function, for which the derivative is not defined.
- candiodari 9y ago
  > Activation functions introduce nonlinearity so that deep learning methods can hopefully learn things linear methods can't. I would just like to clarify: deep learning methods definitely DO learn things that linear methods can't. Using linear functions, no matter how many layers, essentially boils down to the "perceptron" architecture. You can Google it and they will be mostly talking about it's limitations (for instance, it's famously unable to learn XOR, Google "perceptron XOR", essentially the issue is that XOR is not linearly separable, so it can only be expressed as a nonlinear function). You can give a simple proof for this. If you examine what a neural network layer (without activation function) does, in matrix terms. You take the input vector X (1xn), the layer weights W (nxm), and the output vector O (1xm). Then computing the layer output is simply O = WX. Now we can envision what happens with two layers. W (mxn) and V(mxn). The output of running two layers then becomes: O = V(WX) However, matrix multiplication is associative: O = V(WX) = (VW)X, and VW is just a matrix So for every 2 layer linear neural network there is a 1 layer neural network that gives the exact same result. So there's no reason to have 2 layers. A famous result proving this is the "deep" networks (with activation funcions) are universal function approximators (explained here: [1]). Note that this "deep" should be understood in the 1995 meaning of deep neural networks, which is essentially "at least 2 layers", and usually "exactly 2 layers", not the 2010+ one where one means 10 to 200 layer deep networks. [1] http://neuralnetworksanddeeplearning.com/chap4.html http://neuralnetworksanddeeplearning.com/chap4.html
iaw 9y ago
There are two approaches to this type of problem, bottom-up and top-down: Bottom-up is the preferred solution because you can derive everything from earlier principles, math is always bottoms up. Physics tries to be but occasionally fails. Top-down is investigatory, you have something that exists and you want to understand it. Chemistry, biology, and genetics are excellent examples of systems that we can't derive from first principles that we've done well with using the top-down approach. Charting the space of the meta-problem (this config --> this behavior) is a valuable first step in better understanding these structures even if we can never derive their behavior.

alexnewman 9y ago

Is that the only problem? Also why make a network of a certain length. Why not farther. Depth and it’s relation to the degree of the estimated polynomial is well undertood. What’s less understood is why not so deep

inputcoffee 9y ago
I didn't mean to say that it is the only problem. It is more like the "meta problem" as far as theory is concerned. There are many parameters (layers, nodes, connectivity, drop off, number of trials etc). Right now they are guessed at by trial and error. The thing preventing you from saying, "Oh, just reduce the layers by 1, and add more nodes" is the non-linearity.

bitL 9y ago

Why is the lack of theory a problem? At some point we have to accept some problems are "out of our league" and use whatever is available even without fully understanding it. We can't even understand simple specializations in computer vision, yet we expect to understand a more general method? It's not like 15 years ago there weren't math theorems proven by enumerating them on a computer. I understand it breaks psyche and pride of some scientists, so what? Universe can't be expected to fit into humanity's collective brain.

mikebenfield 9y ago
Because it's much easier to work with and improve something if there's a theory behind it? There's a spectrum between "just keep trying tons of crap and see what works" and "do this simple, well-understood calculation to see exactly what will work." Is it not obvious why it's nicer to be on the latter end of the spectrum than the former?
- bitL 9y ago
  Sure, but currently deep learning is more like experimental physics. You try stuff and see what works and empirically improve your understanding. Then you can generalize some heuristics from this and use that "recipe" in the future. You figured out ReLU suddenly made something work, yet Swish turned out better so you can forget about ReLU now. And as you can treat deep learning (in supervised mode) as non-linear optimization, I doubt we'll come up with a proper theory unless P=NP. We can't even understand far simpler non-linear optimization problems, not to mention ones which can be arbitrarily parametrized in 2nd order...
  - mturmon 9y ago
    But to just restate the comment you're replying to, clearly theory can prune a lot of branches on the "iterative experiment-based design refinement" method you are proposing. Also, I'll mention that you're being too pessimistic about what theory can accomplish. The learning problem is much more constrained than P = NP. The use, for example, of a training set of N examples, drawn iid, and evaluation on samples drawn from the same distribution imposes a lot of structure. I don't know if you're familiar with VC theory, but it's an example of the kind of "surprising" guarantees that can be derived in this setting. Other general examples are weak learning, the bias/variance tradeoff, and (in SVMs) the notion of large margin classifiers. An applicable "theory" of design is what separates engineering from just mucking around.
  - enord 9y ago
    Inscrutable models? Check. "I know it when i see it" success criteria? Check. Unjustified optimism? Check. Sounds more like alchemy than experimental physics to me.

mikebenfield 9y ago

> Neural Networks are non-linear because of their activation functions. You need a differentiable function in order to take the derivative so you can back-prop the error, more or less. To clarify: linear functions would still be differentiable without an activation function. The problem is that the composition of linear functions is just another linear function, so you gain nothing by having multiple layers; you might as well just do some kind of linear regression or classification. Activation functions introduce nonlinearity so that deep learning methods can hopefully learn things linear methods can't.

inputcoffee 9y ago
When I wrote that, I was thinking of the unit step function, for which the derivative is not defined.
candiodari 9y ago
> Activation functions introduce nonlinearity so that deep learning methods can hopefully learn things linear methods can't. I would just like to clarify: deep learning methods definitely DO learn things that linear methods can't. Using linear functions, no matter how many layers, essentially boils down to the "perceptron" architecture. You can Google it and they will be mostly talking about it's limitations (for instance, it's famously unable to learn XOR, Google "perceptron XOR", essentially the issue is that XOR is not linearly separable, so it can only be expressed as a nonlinear function). You can give a simple proof for this. If you examine what a neural network layer (without activation function) does, in matrix terms. You take the input vector X (1xn), the layer weights W (nxm), and the output vector O (1xm). Then computing the layer output is simply O = WX. Now we can envision what happens with two layers. W (mxn) and V(mxn). The output of running two layers then becomes: O = V(WX) However, matrix multiplication is associative: O = V(WX) = (VW)X, and VW is just a matrix So for every 2 layer linear neural network there is a 1 layer neural network that gives the exact same result. So there's no reason to have 2 layers. A famous result proving this is the "deep" networks (with activation funcions) are universal function approximators (explained here: [1]). Note that this "deep" should be understood in the 1995 meaning of deep neural networks, which is essentially "at least 2 layers", and usually "exactly 2 layers", not the 2010+ one where one means 10 to 200 layer deep networks. [1] http://neuralnetworksanddeeplearning.com/chap4.html http://neuralnetworksanddeeplearning.com/chap4.html

iaw 9y ago

There are two approaches to this type of problem, bottom-up and top-down: Bottom-up is the preferred solution because you can derive everything from earlier principles, math is always bottoms up. Physics tries to be but occasionally fails. Top-down is investigatory, you have something that exists and you want to understand it. Chemistry, biology, and genetics are excellent examples of systems that we can't derive from first principles that we've done well with using the top-down approach. Charting the space of the meta-problem (this config --> this behavior) is a valuable first step in better understanding these structures even if we can never derive their behavior.