9 ms·

Can I get an ELI5 on how physical neurons, stuck in a measly 3 dimensions, can possibly form higher-dimensional connections on a large scale? I understand high

Can I get an ELI5 on how physical neurons, stuck in a measly 3 dimensions, can possibly form higher-dimensional connections on a large scale? I understand higher dimensional connections in theory (such as in an abstract representation of neurons within a computer), but I can’t imagine how more highly-connected neurons could all physically fit together in meat space.

cochne 5y ago
Consider three neurons all connected together. Now consider that each of them may have some 'voltage' anywhere between 0 and 1. Using three neurons you could describe boxes of different shapes in three dimensions. Add more and you get whatever large dimension you want.
deleted 5y ago
[deleted]
andyxor 5y ago
see related talk by the first author: "Dynamic representations reduce interference in short-term memory": https://www.youtube.com/watch?v=uy7BUzcAenw https://www.youtube.com/watch?v=uy7BUzcAenw
ajuc 5y ago
> Can I get an ELI5 on how physical neurons, stuck in a measly 3 dimensions, can possibly form higher-dimensional connections on a large scale? You can multiplex in frequency and time. I'm not sure if neurons do it, but it's certainly possible with computer networks.
wyager 5y ago
Your stick of RAM is also stuck in 3 dimensions but it reifies a, say, 32-billion-dimensional vector over Z/2Z.
CuriouslyC 5y ago
If you take a matrix of covariance or similarity between neurons based on firing pattern, and try to reduce it to the sum of a weighted set of vectors, the number of vectors you would need to accurately model the system gives you the dimensionality of the space.
- fao_ 5y ago
  This does not seem particularly like an "Explain Like I'm 5"-parsable comment that the posted asked for.
  - dogma1138 5y ago
    This isn’t about the 3 dimensional structure the neurons occupy, but about their operational degrees of freedom. Think about how a CNC machine works, you can have CNC with more than 3 axis, for example a 4 axis CNC machine can move left/right up/down backwards/forwards and also have another axis which can rotate in a given plane. From a more mathematical perspective just think about the number of parameters in a system (excluding reduction) each parameter would be a dimension.
    - anu7df 5y ago
      Appreciate the attempt, but in this example the 4th axis is not independent since the motion along that axis can be achieved, with some complexity, by the motion along the other axes. Granted this is not very useful for a machinist because it will be very tedious to machine a part this way compared to the dedicated 4th rotating axis, but mathematically it is redundant. I have found it easiest to think of a logical dimensions or configurations when thinking of higher dimensions. Physically it can be a row of bulbs (lighted or not) wherein N bulbs (dimensions) can represent 2^n states in total. The 2 here can be increased by having bulbs that can light up in many colours.
      - posterboy 5y ago
        its not redundant. without rotation it could only ever drill downwards. Smartphones eg. measure six dimensions of freedome, including rotation about every axis. 3 for location, 3 for orientation. this has very little to do with synapses.
  - hitlerism 5y ago
    The vectors are in a configuration vector space, not a physical vector space.
fsociety 5y ago
Think of it less as n-dimensional in meat space and more of n-dimensional in how it functions.
exporectomy 5y ago
Do you mean due to the thickness of each connection, they would occupy too much space if the number of dimensions was too high? Not necessarily 4 or more, just very high because there are on the order of n^2 connections for n neurons? In the visual cortex, neurons are arranged in layers of 2D sheets, so that perhaps gives an extra dimension to fit connections between layers.
dopu 5y ago
If I’m recording from N neurons, I’m recording from an N-dimensional system. Each neuron’s firing rate is an axis in this space. If each neuron is maximally uncorrelated from all other neurons, the system will be maximally high dimensional. Its dimensionality will be N. Geometrically, you can think of the state vector of the system (where again, each element is the firing rate of one neuron) as eventually visiting every part of this N-dimensional space. Interestingly, however, neural activity actually tends to be fairly low dimensional (3, 4, 5 dimensional) across most experiments we’ve recorded from. This is because neurons tend to be highly correlated with each other. So the state vector of neural activity doesn’t actually visit every point in this high dimensional space. It tends to stay in a low dimensional space, or on a “manifold” within the N-dimensional space.
- chadcmulligan 5y ago
  Would you have any further reading on this? Sounds fascinating.
  - dopu 5y ago
    Agreed, it's really cool :). A lot of this is very new -- it's only been in the past decade and a half or so that we've been able to record from large populations of neurons (on the order of hundreds and up, see [0]). But there are a lot of smart people working on figuring out how to make sense of this data, and why we see low-dimensional signals in these population recordings. Here are some good reviews on the subject: [1], [2], [3], [4], and [5]. [0]: https://stevenson.lab.uconn.edu/scaling/ https://stevenson.lab.uconn.edu/scaling/ [1]: https://www.nature.com/articles/nn.3776 https://www.nature.com/articles/nn.3776 [2]: https://doi.org/10.1016/j.conb.2015.04.003 https://doi.org/10.1016/j.conb.2015.04.003 [3]: https://doi.org/10.1016/j.conb.2019.02.002 https://doi.org/10.1016/j.conb.2019.02.002 [4]: https://arxiv.org/abs/2104.00145 https://arxiv.org/abs/2104.00145 [5]: https://doi.org/10.1016/j.neuron.2017.05.025 https://doi.org/10.1016/j.neuron.2017.05.025
    - trott 5y ago
      I'm curious about how much of this apparent low dimensionality is explained by (1) the physical proximity of the neurons being recorded, (2) poverty of the stimuli (just 4 sequences in this paper, if I'm not mistaken)
dboreham 5y ago
Same as a silicon chip stuck in 2 dimensions can.
deleted 5y ago
[deleted]
ww520 5y ago
The vector here refers to the "feature vector" where the dimension is the number of elements in the vector. E.g. a feature vector of [size, length, width, height, color, shape, smell] has 7 dimensions. A feature vector for the space has 3 dimensions [x, y, z]. The term "higher dimension" just means the number of features encoded in the vector is higher than usual. In the context of neurons, while the neurons are in the 3 spatial dimensions, the connections of each neuron can be encoded in a feature vector. Each connection can specialize on one feature, e.g. the hair color of the person. These connection features can be encoded in a vector. The number of connections becomes the dimension of the vector. Not to be confused with the physical 3D spatial dimensions of the neurons. The nice thing about encoding things in vectors is that you can use generic math to manipulate them. E.g. rotation mentioned in this article, orthogonality of vectors implies they have no overlap, or dot product of vectors measures how "similar" they are. Apparently this article shows that different versions of the sensory data encoded in neurons can be rotated just like vector rotation so that they are orthogonal and won't interfere with each other. Linear algebra usually deals with 2 or 3 dimensions. Geometric algebra works better on higher dimension vectors.
Salgat 5y ago
Don't conflate physical and logical, in this case we don't care about the physical dimensions, only how the logic is expressed. Even a 2D function can be expressed in N-dimensional parameters, such as y = a1 * x + a2 * x^2 + a3 * x^3 + a4 * x^4 where you only have one input and one output, but 4 constants that can be adjusted. These 4 constants make up a 4D vector.
austinjp 5y ago
This is fun, I'm enjoying reading the replies :) I'm certainly no expert, but attempting an explanation helps me exercise my personal understanding, so here goes. Corrections welcome. The "connections" you mention aren't the issue, in my understanding of the biology. Neurons are already very strongly interconnected by numerous synapses, so they already do physically fit together in their available 3D space, and appear capable of representing high-dimensional concepts. (See caveat below.) The "higher dimensions" here are not where the neurons exist, only what they're capable of representing. If we think about a representation of the concept of a "dog" for example, there are many dimensions. Size, colour, breed, temperament, barking, growling, panting, etc etc. Those attributes are dimensions. Take two dog attributes: size and breed. You can plot a graph of dogs, each dog being a mark on the graph of size vs breed. Add a third dimension and turn the graph into a cube: temperament. You can probably imagine plotting dogs inside this three dimensional space. It's very difficult to imagine that graph extending into 4th, 5th or further dimensions. And yet, you can easily imagine, say, a dog that's a large, black, friendly Labrador with a deep bark who growls only rarely. We could say that dog can be represented as a point in 6-dimensional space (or perhaps a 6-dimensional slice through a space with even more dimensions, just a slice through 3D space could produce a 2D graph). The number of connections between neurons may be related to the number of dimensions they can represent. In honesty, I don't know, and I guess that if there is a relationship it may not be linear. So neurons might be capable of representing 4 dimensions with fewer than 4 synapses, for example, I don't know. Seems possible to me, though. Caveat: I think my reasoning here may be fallacious: "the fact that neurons are capable of representing high-dimension concepts demonstrates that they have adequate synapses to do so". It seems akin to anthropocentrism, I'm not sure. Perhaps it's just a circular argument. I think it provides an adequate basis for an ELI5 though. I look forward to further comments!
posterboy 5y ago
the ELI 5 of higher dimensions explained mathematically in text is that a coordinate in R^3 is identified uniquely by a three tuple u = (x, y, z). A four touple simply adds one dimension. That might be a time coordinate, color, etc. If I remember correctly, the integers Z form spaces, too. Z^2 can be illustrated as grid, where every node is uniquely identified again coordinates or by two of its neighbours, eitherway v = (a, b). Adjency lists or index matrices are common ways to encode graphs. My modelnof a neuron network is then a graph. I imagine that, since Neurons have many more Synapses, that's how you get a manifold with many more coordinates. Each Neuron stores action potential much like color of a pixel and its state evolves over time, but that's when the model becomes limited. How it actually represents complex information in this structure I don't know. PS: Or very simply put, physics has more than three dimensions.

cochne 5y ago

Consider three neurons all connected together. Now consider that each of them may have some 'voltage' anywhere between 0 and 1. Using three neurons you could describe boxes of different shapes in three dimensions. Add more and you get whatever large dimension you want.

deleted 5y ago

[deleted]

andyxor 5y ago

see related talk by the first author: "Dynamic representations reduce interference in short-term memory": https://www.youtube.com/watch?v=uy7BUzcAenw https://www.youtube.com/watch?v=uy7BUzcAenw

ajuc 5y ago

> Can I get an ELI5 on how physical neurons, stuck in a measly 3 dimensions, can possibly form higher-dimensional connections on a large scale? You can multiplex in frequency and time. I'm not sure if neurons do it, but it's certainly possible with computer networks.

wyager 5y ago

Your stick of RAM is also stuck in 3 dimensions but it reifies a, say, 32-billion-dimensional vector over Z/2Z.

CuriouslyC 5y ago

If you take a matrix of covariance or similarity between neurons based on firing pattern, and try to reduce it to the sum of a weighted set of vectors, the number of vectors you would need to accurately model the system gives you the dimensionality of the space.

fao_ 5y ago
This does not seem particularly like an "Explain Like I'm 5"-parsable comment that the posted asked for.
- dogma1138 5y ago
  This isn’t about the 3 dimensional structure the neurons occupy, but about their operational degrees of freedom. Think about how a CNC machine works, you can have CNC with more than 3 axis, for example a 4 axis CNC machine can move left/right up/down backwards/forwards and also have another axis which can rotate in a given plane. From a more mathematical perspective just think about the number of parameters in a system (excluding reduction) each parameter would be a dimension.
  - anu7df 5y ago
    Appreciate the attempt, but in this example the 4th axis is not independent since the motion along that axis can be achieved, with some complexity, by the motion along the other axes. Granted this is not very useful for a machinist because it will be very tedious to machine a part this way compared to the dedicated 4th rotating axis, but mathematically it is redundant. I have found it easiest to think of a logical dimensions or configurations when thinking of higher dimensions. Physically it can be a row of bulbs (lighted or not) wherein N bulbs (dimensions) can represent 2^n states in total. The 2 here can be increased by having bulbs that can light up in many colours.
    - posterboy 5y ago
      its not redundant. without rotation it could only ever drill downwards. Smartphones eg. measure six dimensions of freedome, including rotation about every axis. 3 for location, 3 for orientation. this has very little to do with synapses.
- hitlerism 5y ago
  The vectors are in a configuration vector space, not a physical vector space.

fsociety 5y ago

Think of it less as n-dimensional in meat space and more of n-dimensional in how it functions.

exporectomy 5y ago

Do you mean due to the thickness of each connection, they would occupy too much space if the number of dimensions was too high? Not necessarily 4 or more, just very high because there are on the order of n^2 connections for n neurons? In the visual cortex, neurons are arranged in layers of 2D sheets, so that perhaps gives an extra dimension to fit connections between layers.

dopu 5y ago

If I’m recording from N neurons, I’m recording from an N-dimensional system. Each neuron’s firing rate is an axis in this space. If each neuron is maximally uncorrelated from all other neurons, the system will be maximally high dimensional. Its dimensionality will be N. Geometrically, you can think of the state vector of the system (where again, each element is the firing rate of one neuron) as eventually visiting every part of this N-dimensional space. Interestingly, however, neural activity actually tends to be fairly low dimensional (3, 4, 5 dimensional) across most experiments we’ve recorded from. This is because neurons tend to be highly correlated with each other. So the state vector of neural activity doesn’t actually visit every point in this high dimensional space. It tends to stay in a low dimensional space, or on a “manifold” within the N-dimensional space.

chadcmulligan 5y ago
Would you have any further reading on this? Sounds fascinating.
- dopu 5y ago
  Agreed, it's really cool :). A lot of this is very new -- it's only been in the past decade and a half or so that we've been able to record from large populations of neurons (on the order of hundreds and up, see [0]). But there are a lot of smart people working on figuring out how to make sense of this data, and why we see low-dimensional signals in these population recordings. Here are some good reviews on the subject: [1], [2], [3], [4], and [5]. [0]: https://stevenson.lab.uconn.edu/scaling/ https://stevenson.lab.uconn.edu/scaling/ [1]: https://www.nature.com/articles/nn.3776 https://www.nature.com/articles/nn.3776 [2]: https://doi.org/10.1016/j.conb.2015.04.003 https://doi.org/10.1016/j.conb.2015.04.003 [3]: https://doi.org/10.1016/j.conb.2019.02.002 https://doi.org/10.1016/j.conb.2019.02.002 [4]: https://arxiv.org/abs/2104.00145 https://arxiv.org/abs/2104.00145 [5]: https://doi.org/10.1016/j.neuron.2017.05.025 https://doi.org/10.1016/j.neuron.2017.05.025
  - trott 5y ago
    I'm curious about how much of this apparent low dimensionality is explained by (1) the physical proximity of the neurons being recorded, (2) poverty of the stimuli (just 4 sequences in this paper, if I'm not mistaken)

dboreham 5y ago

Same as a silicon chip stuck in 2 dimensions can.

deleted 5y ago

[deleted]

ww520 5y ago

The vector here refers to the "feature vector" where the dimension is the number of elements in the vector. E.g. a feature vector of [size, length, width, height, color, shape, smell] has 7 dimensions. A feature vector for the space has 3 dimensions [x, y, z]. The term "higher dimension" just means the number of features encoded in the vector is higher than usual. In the context of neurons, while the neurons are in the 3 spatial dimensions, the connections of each neuron can be encoded in a feature vector. Each connection can specialize on one feature, e.g. the hair color of the person. These connection features can be encoded in a vector. The number of connections becomes the dimension of the vector. Not to be confused with the physical 3D spatial dimensions of the neurons. The nice thing about encoding things in vectors is that you can use generic math to manipulate them. E.g. rotation mentioned in this article, orthogonality of vectors implies they have no overlap, or dot product of vectors measures how "similar" they are. Apparently this article shows that different versions of the sensory data encoded in neurons can be rotated just like vector rotation so that they are orthogonal and won't interfere with each other. Linear algebra usually deals with 2 or 3 dimensions. Geometric algebra works better on higher dimension vectors.

Salgat 5y ago

Don't conflate physical and logical, in this case we don't care about the physical dimensions, only how the logic is expressed. Even a 2D function can be expressed in N-dimensional parameters, such as y = a1 * x + a2 * x^2 + a3 * x^3 + a4 * x^4 where you only have one input and one output, but 4 constants that can be adjusted. These 4 constants make up a 4D vector.

austinjp 5y ago

This is fun, I'm enjoying reading the replies :) I'm certainly no expert, but attempting an explanation helps me exercise my personal understanding, so here goes. Corrections welcome. The "connections" you mention aren't the issue, in my understanding of the biology. Neurons are already very strongly interconnected by numerous synapses, so they already do physically fit together in their available 3D space, and appear capable of representing high-dimensional concepts. (See caveat below.) The "higher dimensions" here are not where the neurons exist, only what they're capable of representing. If we think about a representation of the concept of a "dog" for example, there are many dimensions. Size, colour, breed, temperament, barking, growling, panting, etc etc. Those attributes are dimensions. Take two dog attributes: size and breed. You can plot a graph of dogs, each dog being a mark on the graph of size vs breed. Add a third dimension and turn the graph into a cube: temperament. You can probably imagine plotting dogs inside this three dimensional space. It's very difficult to imagine that graph extending into 4th, 5th or further dimensions. And yet, you can easily imagine, say, a dog that's a large, black, friendly Labrador with a deep bark who growls only rarely. We could say that dog can be represented as a point in 6-dimensional space (or perhaps a 6-dimensional slice through a space with even more dimensions, just a slice through 3D space could produce a 2D graph). The number of connections between neurons may be related to the number of dimensions they can represent. In honesty, I don't know, and I guess that if there is a relationship it may not be linear. So neurons might be capable of representing 4 dimensions with fewer than 4 synapses, for example, I don't know. Seems possible to me, though. Caveat: I think my reasoning here may be fallacious: "the fact that neurons are capable of representing high-dimension concepts demonstrates that they have adequate synapses to do so". It seems akin to anthropocentrism, I'm not sure. Perhaps it's just a circular argument. I think it provides an adequate basis for an ELI5 though. I look forward to further comments!

posterboy 5y ago

the ELI 5 of higher dimensions explained mathematically in text is that a coordinate in R^3 is identified uniquely by a three tuple u = (x, y, z). A four touple simply adds one dimension. That might be a time coordinate, color, etc. If I remember correctly, the integers Z form spaces, too. Z^2 can be illustrated as grid, where every node is uniquely identified again coordinates or by two of its neighbours, eitherway v = (a, b). Adjency lists or index matrices are common ways to encode graphs. My modelnof a neuron network is then a graph. I imagine that, since Neurons have many more Synapses, that's how you get a manifold with many more coordinates. Each Neuron stores action potential much like color of a pixel and its state evolves over time, but that's when the model becomes limited. How it actually represents complex information in this structure I don't know. PS: Or very simply put, physics has more than three dimensions.