4 ms·

I don't think it is a good idea to describe neural networks as a large graph of neurons interacting with each other. It is not really helpful to understand what

I don't think it is a good idea to describe neural networks as a large graph of neurons interacting with each other. It is not really helpful to understand what is going on inside. It is more useful to understand them as a series of transforms that bend and fold the input space, in order to place pairs of similar items close to each other. I would like to see people trying to illustrate that instead. It also has the benefit of making the connection with linear algebra much easier to understand.

farresito 5y ago
Totally agree with you. The article that opened my eyes was this[0] one. This[1] video is also very good. [0] https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ [1] https://www.youtube.com/watch?v=e5xKayCBOeU https://www.youtube.com/watch?v=e5xKayCBOeU
joefourier 5y ago
You really find large n-dimensional transforms easier to reason about and visualise, as opposed to layers of neurons with connections? You don’t find it much more intuitive to see it as a graph once you start adding recurrence, convolutions, sparsity, dropout, connections across multiple layers, etc., let alone coming up with new concepts? I think it’s useful to understand it in both ways, but our intuitions about transforms are largely useless when the number of dimensions is high enough.
- nerdponx 5y ago
  It's good to have both perspectives. Ideally you learn the layers-of-transforms version alongside the styled graph-of-neurons version. If you had to only pick one, which one you learn would depend a lot on what kind of student you are and what your goals are. I think the layers-of-transforms version is "less wrong" in general, but probably harder to understand, so it's maybe better if you had to learn just one.
- farresito 5y ago
  Not the person you are answering to, but I think it's all about the level of abstraction you want to reason at. I didn't grok neural networks until I visualized the transformations that were happening in a very simple network. Once that made sense, I could start thinking in terms of layers.
- ravi-delia 5y ago
  I think understanding how neural networks work is easiest if you think of them as networks. Reasoning about why they work is a lot easier thinking about them as transformations. It's not like you're actually picturing all the parameters of a nontrivial network one way or the other.
mjburgess 5y ago
A Neural Network isnt a graph in any case, and isn't based on the brain. As you said, it's a sequence of transformations. NB. If it's a graph: write out the edge list (etc.) . NNs are diagrammed as graphs, but this is highly misleading.
- Retric 5y ago
  Neural Networks are a graph more specifically a Weighted Directed Graph. They are also very much modeled after the brain, more specifically they originate from a 1943 paper by neurophysiologist Warren McCulloch and mathematician Walter Pitts who described how neurons in the brain might work by modeling a simple neural network. Of course it’s not an accurate model, but it very much is based on early understanding of biological neurons.
  - mjburgess 5y ago
    The 1943 NN compu sci model corresponds to a 1943 brain model, sure. NNs in an ML sense arent this model. What does any modern NN compu-sci model correspond to? You're making a genetic fallacy here: that since the origin of X had property P, so must X. Eg., marriage was historically a legal contract of ownership, so today it is a legal contract of ownership. The brain has nothing akin to back-propagation, nor supervised training. "Neurones" are not "data about house prices". According to any even highly simplified neurological model, a compu sci model of this, would compute with activations in a radically different manner than a dot-product over domain-specific data. They are also not a weighted directed graph. In a sense almost any mathematical function is a weighted graph if one takes edges to be, eg., general function composition; and nodes to be input/output domains of those functions. However to say a "NN is a network" is presumably meant to be more significant than this trivial observation that essentially "all functions" are. In particular everything which distinguishes a NN from any mathematical functions are operations such as supervised training, backprop, regularization, pooling, etc. And these are not expressible as functions within this graph, but they are functions over the graph. So any actual "network diagram" presented isn't a network. Its a bad diagram of a mathematical function which also includes functions over this diagram. In any case, a neural network -- in the ML sense -- is neither neural nor a network. It is a function of a mathematical function that has no isomorphism to anything in the brain.

farresito 5y ago

Totally agree with you. The article that opened my eyes was this[0] one. This[1] video is also very good. [0] https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ [1] https://www.youtube.com/watch?v=e5xKayCBOeU https://www.youtube.com/watch?v=e5xKayCBOeU

joefourier 5y ago

You really find large n-dimensional transforms easier to reason about and visualise, as opposed to layers of neurons with connections? You don’t find it much more intuitive to see it as a graph once you start adding recurrence, convolutions, sparsity, dropout, connections across multiple layers, etc., let alone coming up with new concepts? I think it’s useful to understand it in both ways, but our intuitions about transforms are largely useless when the number of dimensions is high enough.

nerdponx 5y ago
It's good to have both perspectives. Ideally you learn the layers-of-transforms version alongside the styled graph-of-neurons version. If you had to only pick one, which one you learn would depend a lot on what kind of student you are and what your goals are. I think the layers-of-transforms version is "less wrong" in general, but probably harder to understand, so it's maybe better if you had to learn just one.
farresito 5y ago
Not the person you are answering to, but I think it's all about the level of abstraction you want to reason at. I didn't grok neural networks until I visualized the transformations that were happening in a very simple network. Once that made sense, I could start thinking in terms of layers.
ravi-delia 5y ago
I think understanding how neural networks work is easiest if you think of them as networks. Reasoning about why they work is a lot easier thinking about them as transformations. It's not like you're actually picturing all the parameters of a nontrivial network one way or the other.

mjburgess 5y ago

A Neural Network isnt a graph in any case, and isn't based on the brain. As you said, it's a sequence of transformations. NB. If it's a graph: write out the edge list (etc.) . NNs are diagrammed as graphs, but this is highly misleading.

Retric 5y ago
Neural Networks are a graph more specifically a Weighted Directed Graph. They are also very much modeled after the brain, more specifically they originate from a 1943 paper by neurophysiologist Warren McCulloch and mathematician Walter Pitts who described how neurons in the brain might work by modeling a simple neural network. Of course it’s not an accurate model, but it very much is based on early understanding of biological neurons.
- mjburgess 5y ago
  The 1943 NN compu sci model corresponds to a 1943 brain model, sure. NNs in an ML sense arent this model. What does any modern NN compu-sci model correspond to? You're making a genetic fallacy here: that since the origin of X had property P, so must X. Eg., marriage was historically a legal contract of ownership, so today it is a legal contract of ownership. The brain has nothing akin to back-propagation, nor supervised training. "Neurones" are not "data about house prices". According to any even highly simplified neurological model, a compu sci model of this, would compute with activations in a radically different manner than a dot-product over domain-specific data. They are also not a weighted directed graph. In a sense almost any mathematical function is a weighted graph if one takes edges to be, eg., general function composition; and nodes to be input/output domains of those functions. However to say a "NN is a network" is presumably meant to be more significant than this trivial observation that essentially "all functions" are. In particular everything which distinguishes a NN from any mathematical functions are operations such as supervised training, backprop, regularization, pooling, etc. And these are not expressible as functions within this graph, but they are functions over the graph. So any actual "network diagram" presented isn't a network. Its a bad diagram of a mathematical function which also includes functions over this diagram. In any case, a neural network -- in the ML sense -- is neither neural nor a network. It is a function of a mathematical function that has no isomorphism to anything in the brain.