8 ms·
Something to keep in mind though is that in a high-dimensional space, approximate orthogonality of independent vectors is almost guaranteed.
by trott 5y ago
Something to keep in mind though is that in a high-dimensional space, approximate orthogonality of independent vectors is almost guaranteed.
- filoeleven 5y agoCan you say a bit more on what that means in this context?
- FigmentEngine 5y agoprobably a reference to the curse of dimensionality
- fighterpilot 5y agoNot sure about the neuroscience context, but if you have two large ("high-dimensional") vectors of variables that have a population correlation of zero ("independent"), then the dot product of a sample is likely to be close to zero ("orthogonal") due to the law of large numbers.
- adampk 5y agoDo you mean to say that the neurons in the brain are operating in a higher-dimensional space than 3?
- frisco 5y agoYes definitely. Here the "space" doesn't refer to physical space, but an abstract vector space that neuron's tuning represents. For example, there is a famous paper[1] that showed neurons could be responsive to abstract concepts -- for example, one might fire for "Bill Clinton" regardless of whether the stimulus is a photo of him, his name written as letters, or even (with weaker activation) photos/text of other members of his family or other concepts adjacent to him. The neuron's activity gives a vector in this high dimensional concept space, and that's the "space" GP is referring to. [1] https://www.nature.com/articles/nature03687 https://www.nature.com/articles/nature03687
- PullJosh 5y agoCan I get an ELI5 on how physical neurons, stuck in a measly 3 dimensions, can possibly form higher-dimensional connections on a large scale? I understand higher dimensional connections in theory (such as in an abstract representation of neurons within a computer), but I can’t imagine how more highly-connected neurons could all physically fit together in meat space.
- cochne 5y agoConsider three neurons all connected together. Now consider that each of them may have some 'voltage' anywhere between 0 and 1. Using three neurons you could describe boxes of different shapes in three dimensions. Add more and you get whatever large dimension you want.
- deleted 5y ago[deleted]
- andyxor 5y agosee related talk by the first author: "Dynamic representations reduce interference in short-term memory": https://www.youtube.com/watch?v=uy7BUzcAenw https://www.youtube.com/watch?v=uy7BUzcAenw
- ajuc 5y ago> Can I get an ELI5 on how physical neurons, stuck in a measly 3 dimensions, can possibly form higher-dimensional connections on a large scale? You can multiplex in frequency and time. I'm not sure if neurons do it, but it's certainly possible with computer networks.
- wyager 5y agoYour stick of RAM is also stuck in 3 dimensions but it reifies a, say, 32-billion-dimensional vector over Z/2Z.
- CuriouslyC 5y agoIf you take a matrix of covariance or similarity between neurons based on firing pattern, and try to reduce it to the sum of a weighted set of vectors, the number of vectors you would need to accurately model the system gives you the dimensionality of the space.
- deleted 5y ago[deleted]
- gleenn 5y agoIf you think of groups of neurons in arbitrary dimensions, where some groups fire together for some things, and a different group with some overlap fire for other things, then it's like two dimensions where a line is a sense or thought and the lines are crossing where they fire for both memories. So two thoughts along two dimensions can cross and light up that subset of neurons. If the two thoughts, or lines, are orthogonal, then not many neurons are both firing for thoughts. If you have many many neurons, and many many memories, then the dimensionality, or possible subsets of firing neurons, is huge. Like our two lines but now in three dimensions, there are a lot of ways for them not to overlap. So the possibility that many things in that space are orthogonal is likely. In a highly dimensional space, a whole lot of things don't overlap.
- andyxor 5y agoYes, grid cells in the hippocampus [0] form a coordinate system that is used for 4D spatiotemporal navigation [1], as well as navigation in abstract high-dimensional "concept space" [2] [0] http://www.scholarpedia.org/article/Grid_cells http://www.scholarpedia.org/article/Grid_cells [1] Time (and space) in the hippocampus https://pubmed.ncbi.nlm.nih.gov/28840180/ https://pubmed.ncbi.nlm.nih.gov/28840180/ [2] Organizing conceptual knowledge in humans with a gridlike code: https://science.sciencemag.org/content/352/6292/1464 https://science.sciencemag.org/content/352/6292/1464
- darwingr 5y agoYes but only in aggregate, like how adding a column to a database table is also adding a "dimension" to said data. I'm not convinced the author's analogy of cross-writing to fit more information on a page is actually going to be helpful to most people's understanding. It led me at least to try to imagine visually what's going on, to picture the input being physically rotated. This is more akin to the more abstract but inclusive concept of rotation from linear algebra, where more dimensions (of information, not space or time) makes sense.
- mrbungie 5y agoYes, just as a set of 1000-levers (arbitrary number, but highly dimensional) can influence a machine (in our 3d reality).
- dopu 5y agoSure, but the neural activity is actually low-dimensional (see Extended Fig 5e). By day 4, the first two principal components of the neural activity explains 75% of the variance in response. ~3-4 dimensions is not particularly high dimensional.
- iandanforth 5y agoNo? If the samples are randomly chosen then you'd expect the cosign similarity to be low, but there's no such assumption here, in fact it's the exact opposite.