3 ms·

>Real neurons don't use calculus and activation functions, nor back-propagation, etc in order to learn. This sounds like a (common) failure to understand how a

by Retra 10y ago

>Real neurons don't use calculus and activation functions, nor back-propagation, etc in order to learn. This sounds like a (common) failure to understand how abstractions work. Bridges don't do calculus, but the bridge builder uses calculus to understand what bridges do use (the laws of nature), and thus the calculus abstraction is used to encode the behavior of bridges. Thus you can model bridges using calculus. Similarly, neurons are modeled by calculus. Abstractions are abstract precisely because they are not the concrete thing they model: they are necessarily approximations. They give us the power to simplify at the cost of gaining the capacity to be wrong. The point being this: you can literally use any abstraction you desire to model anything you like. Some will work better than others, and the better they work, the more closely the structure of your abstraction matches the structure of the concretion being modeled.

pekk 10y ago
If you fit some data with a very flexible function approximator, that does not imply any kind of isomorphism between the function approximator itself and the process generating the data. Some people cannot understand this, and believe that if you can closely fit the output of a process with a neural network that it implies the process itself is in some way related to neural networks.
stuxnet79 10y ago
Very well said. It took me a long time to "get it" but once I crossed that threshold I started viewing different mathematical techniques as hammers, some more suited to modeling certain phenomena than others.
cr0sh 10y ago
Don't get me wrong - I may not fully comprehend the math behind what is going on (as I noted, I have little understanding of calculus, and it is one of my failings that I am working to improve on) - but I do understand the need for the abstraction; I understand that it allows us to model the activation function and workings of a neuron, to a certain level of accuracy. I understand that it may -not- be the same way that a real neuron does it, but that it is close enough and works well enough to be useful. At the same time, I wonder if there isn't a simpler way of doing all of this - if there isn't a simpler model for the abstraction of a neuron that doesn't require back-propagation or calculus? In other words, have we become so use to the current concepts and models of ANNs that we have become hesitant or resistant to imagining other possibilities? Yes - what we have works, and it works well. Honestly, from what I know, and what I have learned (while I may not fully grasp the mathematical and calculus underpinnings of a neural network, I do have a good feel for how both forward and back-prop is implemented), our current general knowledge on ANNs (ie "how to implement a simple neural network") isn't super-complex. I only question whether it could be made simpler (and I'm not talking about a network of perceptrons or RELUs), or if, because of the early work (Pitts and McCulloch mainly), we are going down a less-than-optimum path (to use an ML analogy, we are stuck in a local minima) - one that fortunately works, but maybe there's something better out there? Again - I recognize as a non-expert in this field (I only consider myself a student and hobbyist so far) - I am likely very well off in la-la land. Even so, we know that quite often in engineering, the optimum solution tends to be the simplest solution; sometimes, that solution is simpler than nature (an airplane vs a bird, for instance). I have a feeling that may be the case with ANNs as well. I am not wedded to this idea, though - I just want to put it out there for consideration and maybe discussion. As I noted, what we currently have works well, and isn't a complexity nightmare, and is understandable to anyone willing to understand it. It very well could be it -is- the simplest explanation.

by Retra 10y ago

pekk 10y ago

If you fit some data with a very flexible function approximator, that does not imply any kind of isomorphism between the function approximator itself and the process generating the data. Some people cannot understand this, and believe that if you can closely fit the output of a process with a neural network that it implies the process itself is in some way related to neural networks.

stuxnet79 10y ago

Very well said. It took me a long time to "get it" but once I crossed that threshold I started viewing different mathematical techniques as hammers, some more suited to modeling certain phenomena than others.

cr0sh 10y ago

Don't get me wrong - I may not fully comprehend the math behind what is going on (as I noted, I have little understanding of calculus, and it is one of my failings that I am working to improve on) - but I do understand the need for the abstraction; I understand that it allows us to model the activation function and workings of a neuron, to a certain level of accuracy. I understand that it may -not- be the same way that a real neuron does it, but that it is close enough and works well enough to be useful. At the same time, I wonder if there isn't a simpler way of doing all of this - if there isn't a simpler model for the abstraction of a neuron that doesn't require back-propagation or calculus? In other words, have we become so use to the current concepts and models of ANNs that we have become hesitant or resistant to imagining other possibilities? Yes - what we have works, and it works well. Honestly, from what I know, and what I have learned (while I may not fully grasp the mathematical and calculus underpinnings of a neural network, I do have a good feel for how both forward and back-prop is implemented), our current general knowledge on ANNs (ie "how to implement a simple neural network") isn't super-complex. I only question whether it could be made simpler (and I'm not talking about a network of perceptrons or RELUs), or if, because of the early work (Pitts and McCulloch mainly), we are going down a less-than-optimum path (to use an ML analogy, we are stuck in a local minima) - one that fortunately works, but maybe there's something better out there? Again - I recognize as a non-expert in this field (I only consider myself a student and hobbyist so far) - I am likely very well off in la-la land. Even so, we know that quite often in engineering, the optimum solution tends to be the simplest solution; sometimes, that solution is simpler than nature (an airplane vs a bird, for instance). I have a feeling that may be the case with ANNs as well. I am not wedded to this idea, though - I just want to put it out there for consideration and maybe discussion. As I noted, what we currently have works well, and isn't a complexity nightmare, and is understandable to anyone willing to understand it. It very well could be it -is- the simplest explanation.