3 ms·

Sorry if I'm Mx-splaining -- I don't know your background, but I'm interested in/working on this stuff. Your crumpled space-filling surface makes me think of m

Sorry if I'm Mx-splaining -- I don't know your background, but I'm interested in/working on this stuff. Your crumpled space-filling surface makes me think of manifolds. Machine-learning researchers often speak loosely of e.g. the "manifold"* of possible images of naturalistic scenes embedded within the space of all possible images. The networks are presumably learning a lower-dimensional representation of the world than the dimension of all possible combinations of sense impressions. If you have a 100 pixel by 100 pixel image and each pixel can have 256 intensity levels, then that's 256 to the power of ten thousand possible distinct images. If each image is a point in an abstract space of all possible images, then that space has 256 to the ten thousand dimensions. But the vast, vast, vast majority of the volume of that space corresponds to images that just look like static to humans. So the thinking is that we internally represent images as some learned non-linear transformation to a much lower dimensional set of features that actually correspond to stuff we experience / see. A really simple canonical example is the Swiss Roll dataset. You only need two parameters (numbers) to specify it fully, but you can embed it (nonlinearly) in a 3D space. http://people.cs.uchicago.edu/~dinoj/manifold/swissroll.html http://people.cs.uchicago.edu/~dinoj/manifold/swissroll.html In terms of neuroscience, there are competing ideas (as always). There's a body of work that tries to show that as we learn, the brain encodes our high-dimensional sense-impressions in the lowest possible (most efficient) internal representations. However, some recent work seems to imply that instead the brain uses as high a dimension as possible, but up to some limit that demarcates the transition between being differentiable and not differentiable. They found a beautiful power-law: https://www.biorxiv.org/content/10.1101/374090v1.full https://www.biorxiv.org/content/10.1101/374090v1.full * When ML researchers speak of such a manifold, it's kind of loosey-goosey because a set that includes isolated points that don't have a continuous region around them aren't actually manifolds. In contrast, in the computer graphics and meshing literature people speak of non-manifold geometry which is isolated points and lines that you can't triangulate with 2d or 3d elements. I.e. 1d or 0d elements.

whatshisface 5y ago
>When ML researchers speak of such a manifold, it's kind of loosey-goosey because a set that includes isolated points that don't have a continuous region around them aren't actually manifolds. So they mean "subset" when they say "manifold?"
- JabavuAdams 5y ago
  I suppose so, but they really are trying to convey that all the points lie close to some lower-dimensional crumpled surface. So a subset that was e.g. just a lattice in the high-dimensional space wouldn't fit this mental image. E.g. Generate 3d points that are on a 2d plane +- some small random offset normal to the plane. If the points are isolated, without each having a local neighbourhood, it's not technically a manifold. But, you could describe the data-set as lying on or near a plane to within some tolerance. So they're trying to describe something that is much more specific / strongly constrained than an arbitrary subset, but it doesn't meet the very stringent (and frankly idealized) requirements of a manifold. I wonder whether there is a math-object to describe this? EDIT> Maybe it's just a matter of saying "close to some lower-dimensional manifold", rather than "on a lower-dimensional manifold."
  - whatshisface 5y ago
    >I wonder whether there is a math-object to describe this? Here is how I would phrase it: there is a probability distribution in the higher-dimensional space that expresses P(this image | given that it's a natural image). The level sets of the probability distribution are manifolds.
TheCoreh 5y ago
> Sorry if I'm Mx-splaining -- I don't know your background Not at all! All of this is far more advanced than my knowledge on the topic, and very interesting! Thanks for sharing That idea of representing in the highest dimensionality possible, with some constraint is also very interesting. In that case perhaps the 3D space is being represented in a higher dimensional form that makes it more convenient for some neural processing purpose (e.g. just like we use homogenous coordinates) The 2D-2D case is then just a happy coincidence where the highest representation that makes sense maps 1 to 1 with the actual data.
- JabavuAdams 5y ago
  We should be careful to distinguish between the dimensionality of the physical space, the dimensionality of image data coming in from the retina, and the dimensionality of the navigational representation. Going back to the image example, a 100x100 pixel image is 2D in that it can be shown on a screen, or printed on a page, or laid flat on a plane. But the (abstract) space of all possible images is #intensities_per_pixel to the 100x100 = 10000 power. It's abstract in that each point in this space is not a location in the external world, but specifies one particular image. If you're familiar with phase spaces or configuration spaces from physics, it's like that. The other thing is that we don't seem to have direct access to a 2d or 3d position-tracker sense. So instead, we have to build up some internal representation for navigation, based on our senses which as outlined for images are much higher-dimensional than the physically allowable positions in the world they're sensing. Robotics SLAM is one approach. Then finally, there's the dimensionality of the neural representation itself. Let's say that your internal navigation map is represented by the firing-pattern of a population of neurons (i.e. more than one). Define a time-step, say one millisecond. For simplicity, consider each neuron to just be have an "activity" in the range 0.0 to 1.0 per time-step by e.g. counting the number of spikes emitted per time-step and dividing by some number to get things in a nice range. So now you can represent the activity of a neuron by one number per timestep. If you have a population of 100 neurons, then that's a 100 number string. But ... you can also think of it as a point in a 100 dimensional space. Each point is one particular pattern of neural activations. The entire space is all of the possible patterns of neural activations. Again, this is not a space in the sense of physical positions in the world or in the brain. It's an abstract space. But all the vector space math works. SO we're perceiving 3D by means of retinas that take way more than 3 measurements at an instant, and maybe our brain is finding correlations so that it can represent these in a distributed way in << input_dimensions, but > world_dimensions.

whatshisface 5y ago

>When ML researchers speak of such a manifold, it's kind of loosey-goosey because a set that includes isolated points that don't have a continuous region around them aren't actually manifolds. So they mean "subset" when they say "manifold?"

JabavuAdams 5y ago
I suppose so, but they really are trying to convey that all the points lie close to some lower-dimensional crumpled surface. So a subset that was e.g. just a lattice in the high-dimensional space wouldn't fit this mental image. E.g. Generate 3d points that are on a 2d plane +- some small random offset normal to the plane. If the points are isolated, without each having a local neighbourhood, it's not technically a manifold. But, you could describe the data-set as lying on or near a plane to within some tolerance. So they're trying to describe something that is much more specific / strongly constrained than an arbitrary subset, but it doesn't meet the very stringent (and frankly idealized) requirements of a manifold. I wonder whether there is a math-object to describe this? EDIT> Maybe it's just a matter of saying "close to some lower-dimensional manifold", rather than "on a lower-dimensional manifold."
- whatshisface 5y ago
  >I wonder whether there is a math-object to describe this? Here is how I would phrase it: there is a probability distribution in the higher-dimensional space that expresses P(this image | given that it's a natural image). The level sets of the probability distribution are manifolds.

TheCoreh 5y ago

> Sorry if I'm Mx-splaining -- I don't know your background Not at all! All of this is far more advanced than my knowledge on the topic, and very interesting! Thanks for sharing That idea of representing in the highest dimensionality possible, with some constraint is also very interesting. In that case perhaps the 3D space is being represented in a higher dimensional form that makes it more convenient for some neural processing purpose (e.g. just like we use homogenous coordinates) The 2D-2D case is then just a happy coincidence where the highest representation that makes sense maps 1 to 1 with the actual data.

JabavuAdams 5y ago
We should be careful to distinguish between the dimensionality of the physical space, the dimensionality of image data coming in from the retina, and the dimensionality of the navigational representation. Going back to the image example, a 100x100 pixel image is 2D in that it can be shown on a screen, or printed on a page, or laid flat on a plane. But the (abstract) space of all possible images is #intensities_per_pixel to the 100x100 = 10000 power. It's abstract in that each point in this space is not a location in the external world, but specifies one particular image. If you're familiar with phase spaces or configuration spaces from physics, it's like that. The other thing is that we don't seem to have direct access to a 2d or 3d position-tracker sense. So instead, we have to build up some internal representation for navigation, based on our senses which as outlined for images are much higher-dimensional than the physically allowable positions in the world they're sensing. Robotics SLAM is one approach. Then finally, there's the dimensionality of the neural representation itself. Let's say that your internal navigation map is represented by the firing-pattern of a population of neurons (i.e. more than one). Define a time-step, say one millisecond. For simplicity, consider each neuron to just be have an "activity" in the range 0.0 to 1.0 per time-step by e.g. counting the number of spikes emitted per time-step and dividing by some number to get things in a nice range. So now you can represent the activity of a neuron by one number per timestep. If you have a population of 100 neurons, then that's a 100 number string. But ... you can also think of it as a point in a 100 dimensional space. Each point is one particular pattern of neural activations. The entire space is all of the possible patterns of neural activations. Again, this is not a space in the sense of physical positions in the world or in the brain. It's an abstract space. But all the vector space math works. SO we're perceiving 3D by means of retinas that take way more than 3 measurements at an instant, and maybe our brain is finding correlations so that it can represent these in a distributed way in << input_dimensions, but > world_dimensions.