4 ms·

A Neural Network isnt a graph in any case, and isn't based on the brain. As you said, it's a sequence of transformations. NB. If it's a graph: write out the e

A Neural Network isnt a graph in any case, and isn't based on the brain. As you said, it's a sequence of transformations. NB. If it's a graph: write out the edge list (etc.) . NNs are diagrammed as graphs, but this is highly misleading.

Retric 5y ago
Neural Networks are a graph more specifically a Weighted Directed Graph. They are also very much modeled after the brain, more specifically they originate from a 1943 paper by neurophysiologist Warren McCulloch and mathematician Walter Pitts who described how neurons in the brain might work by modeling a simple neural network. Of course it’s not an accurate model, but it very much is based on early understanding of biological neurons.
- mjburgess 5y ago
  The 1943 NN compu sci model corresponds to a 1943 brain model, sure. NNs in an ML sense arent this model. What does any modern NN compu-sci model correspond to? You're making a genetic fallacy here: that since the origin of X had property P, so must X. Eg., marriage was historically a legal contract of ownership, so today it is a legal contract of ownership. The brain has nothing akin to back-propagation, nor supervised training. "Neurones" are not "data about house prices". According to any even highly simplified neurological model, a compu sci model of this, would compute with activations in a radically different manner than a dot-product over domain-specific data. They are also not a weighted directed graph. In a sense almost any mathematical function is a weighted graph if one takes edges to be, eg., general function composition; and nodes to be input/output domains of those functions. However to say a "NN is a network" is presumably meant to be more significant than this trivial observation that essentially "all functions" are. In particular everything which distinguishes a NN from any mathematical functions are operations such as supervised training, backprop, regularization, pooling, etc. And these are not expressible as functions within this graph, but they are functions over the graph. So any actual "network diagram" presented isn't a network. Its a bad diagram of a mathematical function which also includes functions over this diagram. In any case, a neural network -- in the ML sense -- is neither neural nor a network. It is a function of a mathematical function that has no isomorphism to anything in the brain.
  - Retric 5y ago
    Back propagation etc isn’t a property of a NN it’s a means to generate a NN. You can argue there is some quantitative difference between modern neutral networks but they hadn’t reached the point of trying to train a neural network, it was much more abstract. Also, the brain does actually use a form of supervised learning. Pain for example is effectively a form of input labeling and lack of pain the wanted output.
    - mjburgess 5y ago
      If the neural network is just the piece-wise linear regression function that the NN algorithm generates -- then its just a network like any other mathematical function. And it's a pretty crappy mathematical function, with zero homology to the brain. Nothing in the brain is piece-wise linear.
      - Retric 5y ago
        NN are a more than just feed forward NN. Assuming infinite precision finite NN can be turing complete.
        mjburgess 5y ago
        So is backprop a property or not? > Back propagation etc isn’t a property of a NN it’s a means to generate a NN. You are the one claiming it isnt
        Retric 5y ago
        No, I don’t think you understand the terminology. Feed forward NN means a NN without loops. Back propagation is a method of generating a new NN that only works on a subset of NN. I can generate a random number from a rolling a size sided dice once, just not any possible number.
        mjburgess 5y ago
- jonnycomputer 5y ago
  Yeah, I very much don't understand OP's argument. And its trivial to write out the nodes and edges (at least for trivially sized neural networks).
  - zwaps 5y ago
    I think what op means is this: A graph is mathematically a set of vertices (nodes) and a set of ordered or unordered tuples giving the edges (ties). Now, sometimes you might have a weight on these edges, for example by specifying some function on the edge set. However, it is difficult to see how a neural network that includes operations like sum, multiply and tanh, might be modeled this way. How do you describe a dropout as a graph? I think the argument is that a graph is not sufficient to describe a NN, so technically speaking a NN is not a graph. It is more. It has edges between x and f(x), but we also need to specify what f(x) is. The mathematical definition of a graph doesn't do that.
    - deleted 5y ago
      [deleted]
    - Retric 5y ago
      A weighted graph can include weights for both the vertices and edges. For example a network latency diagram may include the physical wires as separate from the router as the router latency may depend on network load. Similarly routers themselves have internal bandwidth limitations etc. As to the rest separating the NN from everything needed to generate it is a useful distinction. You’re not going to generate a different f(x) by slightly changing the training set etc. It’s however a somewhat arbitrary distinction.
laGrenouille 5y ago
> NB. If it's a graph: write out the edge list (etc.) . I don't understand what issue you are referring to. For a dense network, each pair of adjacent layers forms a complete bipartite graph. In other words, edges are all pairs with one node in layer N and another in layer N+1. CNNs and RNNs take a little more work, but still easy to describe the graph structure.
- zwaps 5y ago
  I think op means that a graph is not sufficient to describe a NN. If a layer is Y=XB, then you draw that as set of nodes Y and individual weight b_ij as edge-weights from X. Right. But can you describe things like concat, max-pooling, attention etc. without changing the meaning of the edges? Or do you have to annotate edges to now mean "apply function here"? If so, op probably wants to say that you are describing more than a graph. There's a graph there, but you need more, you need elaborate descriptions of what edges do. In that case, op could be correct to say that technically, NN are not graphs. Or, perhaps NN can generally be represented by vertices and edge lists. It certainly isn't the usual way to draw them, though.

Retric 5y ago

Neural Networks are a graph more specifically a Weighted Directed Graph. They are also very much modeled after the brain, more specifically they originate from a 1943 paper by neurophysiologist Warren McCulloch and mathematician Walter Pitts who described how neurons in the brain might work by modeling a simple neural network. Of course it’s not an accurate model, but it very much is based on early understanding of biological neurons.

mjburgess 5y ago
The 1943 NN compu sci model corresponds to a 1943 brain model, sure. NNs in an ML sense arent this model. What does any modern NN compu-sci model correspond to? You're making a genetic fallacy here: that since the origin of X had property P, so must X. Eg., marriage was historically a legal contract of ownership, so today it is a legal contract of ownership. The brain has nothing akin to back-propagation, nor supervised training. "Neurones" are not "data about house prices". According to any even highly simplified neurological model, a compu sci model of this, would compute with activations in a radically different manner than a dot-product over domain-specific data. They are also not a weighted directed graph. In a sense almost any mathematical function is a weighted graph if one takes edges to be, eg., general function composition; and nodes to be input/output domains of those functions. However to say a "NN is a network" is presumably meant to be more significant than this trivial observation that essentially "all functions" are. In particular everything which distinguishes a NN from any mathematical functions are operations such as supervised training, backprop, regularization, pooling, etc. And these are not expressible as functions within this graph, but they are functions over the graph. So any actual "network diagram" presented isn't a network. Its a bad diagram of a mathematical function which also includes functions over this diagram. In any case, a neural network -- in the ML sense -- is neither neural nor a network. It is a function of a mathematical function that has no isomorphism to anything in the brain.
- Retric 5y ago
  Back propagation etc isn’t a property of a NN it’s a means to generate a NN. You can argue there is some quantitative difference between modern neutral networks but they hadn’t reached the point of trying to train a neural network, it was much more abstract. Also, the brain does actually use a form of supervised learning. Pain for example is effectively a form of input labeling and lack of pain the wanted output.
  - mjburgess 5y ago
    If the neural network is just the piece-wise linear regression function that the NN algorithm generates -- then its just a network like any other mathematical function. And it's a pretty crappy mathematical function, with zero homology to the brain. Nothing in the brain is piece-wise linear.
    - Retric 5y ago
      NN are a more than just feed forward NN. Assuming infinite precision finite NN can be turing complete.
      - mjburgess 5y ago
        So is backprop a property or not? > Back propagation etc isn’t a property of a NN it’s a means to generate a NN. You are the one claiming it isnt
        Retric 5y ago
        No, I don’t think you understand the terminology. Feed forward NN means a NN without loops. Back propagation is a method of generating a new NN that only works on a subset of NN. I can generate a random number from a rolling a size sided dice once, just not any possible number.
        mjburgess 5y ago
jonnycomputer 5y ago
Yeah, I very much don't understand OP's argument. And its trivial to write out the nodes and edges (at least for trivially sized neural networks).
- zwaps 5y ago
  I think what op means is this: A graph is mathematically a set of vertices (nodes) and a set of ordered or unordered tuples giving the edges (ties). Now, sometimes you might have a weight on these edges, for example by specifying some function on the edge set. However, it is difficult to see how a neural network that includes operations like sum, multiply and tanh, might be modeled this way. How do you describe a dropout as a graph? I think the argument is that a graph is not sufficient to describe a NN, so technically speaking a NN is not a graph. It is more. It has edges between x and f(x), but we also need to specify what f(x) is. The mathematical definition of a graph doesn't do that.
  - deleted 5y ago
    [deleted]
  - Retric 5y ago
    A weighted graph can include weights for both the vertices and edges. For example a network latency diagram may include the physical wires as separate from the router as the router latency may depend on network load. Similarly routers themselves have internal bandwidth limitations etc. As to the rest separating the NN from everything needed to generate it is a useful distinction. You’re not going to generate a different f(x) by slightly changing the training set etc. It’s however a somewhat arbitrary distinction.

laGrenouille 5y ago

> NB. If it's a graph: write out the edge list (etc.) . I don't understand what issue you are referring to. For a dense network, each pair of adjacent layers forms a complete bipartite graph. In other words, edges are all pairs with one node in layer N and another in layer N+1. CNNs and RNNs take a little more work, but still easy to describe the graph structure.

zwaps 5y ago
I think op means that a graph is not sufficient to describe a NN. If a layer is Y=XB, then you draw that as set of nodes Y and individual weight b_ij as edge-weights from X. Right. But can you describe things like concat, max-pooling, attention etc. without changing the meaning of the edges? Or do you have to annotate edges to now mean "apply function here"? If so, op probably wants to say that you are describing more than a graph. There's a graph there, but you need more, you need elaborate descriptions of what edges do. In that case, op could be correct to say that technically, NN are not graphs. Or, perhaps NN can generally be represented by vertices and edge lists. It certainly isn't the usual way to draw them, though.