7 ms·

This can also go wrong, for example visualising probability distributions in low dimensions leads to very wrong intuitions about the behavior of high-dimensiona

This can also go wrong, for example visualising probability distributions in low dimensions leads to very wrong intuitions about the behavior of high-dimensional dimensional distributions.

vbuwivbiu 9y ago
please elaborate - are you thinking of the curse of dimensionality ?
- phreeza 9y ago
  There are many examples, one I came across recently is that the large majority of the probability mass of a high-dimensional gaussian distribution is in a shell at a distance from the mean, the mass at the center is actually quite low. Also anything related to topology, which is important when you are looking at decision boundaries, becomes counterintuitive in high dimensions, because so many things can be adjacent at the same time.
  - shas3 9y ago
    Theoretically, yes. Can you give a more concrete example? Many hard high dimensional and general topology problems can be visualized through their 2D special cases.
  - jstanley 9y ago
    Can you please try and explain why that is? If true, you're very correct that lower-dimensional intuition does not transfer into higher-dimensional spaces: my intuition tells me that a Gaussian distribution drops off as you fall away from the mean, and it's quite easy for me to imagine that in 2 dimensions, 3 dimensions (e.g. by imagining a mound on a plane) and 4 dimensions (e.g. a cloud in 3-space with increased density around the mean). Is my intuition wrong in any of those cases? If so, why? If not, how many dimensions do we need before it becomes wrong?
    - smallnamespace 9y ago
      The center of the distribution always has the highest density, but the ratio of 'probability mass close to centroid' / 'total probability mass' drops off as number of dimensions grows. This is somewhat related to another 'curse of dimensionality' observation, which is that the volume of a hyperball / volume of hyperspace tends towards zero as dimensions grow -- there's just a lot more volume that's in some sense 'far' from the center.
    - orangecat 9y ago
      Because an outlier in any single dimension will put the point outside the "center" of the distribution, and as the number of dimensions increases there's more of a chance of that happening. Say you have an N-dimensional gaussian where each dimension has mean 0 and standard deviation 1. Define the center as the N-dimensional cube whose edges go from -3 to +3 in each dimension. A normally distributed value is within 3 standard deviations of the mean with probability 0.9973, so the probability that an N-dimensional point being in the center is 0.9973^N. With N=4 that's 0.989 which matches your intuition, but at N=1000 it's 0.067 and at N=10000 it's 1.81e-12.
    - nabla9 9y ago
      > it's quite easy for me to imagine that in 2 dimensions It starts to fail really badly when dimension grows. Two simple examples: 1) Consider 3 dimensional unit sphere centered at origin and unit cube centered at origin. Cube is clearly completely inside the sphere. Now generalize to n-dimensions. Hyperdimensional volume of hypercube with side length 1 moves almost completely outside the n-sphere with radius 1 when n-grows. 2) Alternatively almost all volume of n-sphere is close to the surface. These are all very counterintuitive, yet simple to check toy examples. When you start to integrate over more complex multidimensional function, things get weird really fast.
      - tzahola 9y ago
        >Alternatively almost all volume of n-sphere is close to the surface. How does this go against intuition? Intuition from 1/2/3d tells me that the volume of an N-ball is O(r^N), and indeed it is the case in higher dimensions. Therefore it’s easy to see that the difference between the volume of an N-ball of radius r and an N-ball of radius (r + epsilon) will grow exponentially with N.
  - sevenfive 9y ago
    Yeah, but the distance from the center is itself just a one-dimensional gaussian...
  - alan-crowe 9y ago
    This is the problem that makes my brain melt when I try to think about genetics and mutational load. The naive idea is that a species, S, has a correct genome, G, but mutations build up, increasing with each generation. Presumably mutations build up until they are common enough that back-mutations are a thing. Then there is an equilibrium. In a fecund species, each individual has many children, but most have a higher mutational load, many have the save mutational load, and a lucky few have a smaller mutational load, closer to G, the correct genome. Differential reproductive success then maintains the equilibrium. I don't see how the numbers are supposed to work out for large mammals, with each female having under a dozen offspring. To have a decent chance of a back-mutation, the typical member of the species would need one twelfth of their genome to be deleterious mutations. Meanwhile, people are thinking about using CRISPR to correct the human genome, creating unusually happy, healthy people. The underlying thought is that the correct genome is best. But why do we think that the correct genome works at all? Most of the population is in a shell at a distance from the correct genome, the number at the center is actually quite low. Given the combinatorics, with two to the millions of possible genomes, but populations in the millions, the number at the center, or even close, is actually zero. Maybe the correct genome codes for a sickly, miserable individual? My current guess is that the evolution of large mammals with few offspring is constrained by genetic load considerations. It is not sufficient, (or even necessary) for the correct genome to be any good. There needs to be a big blob of mediocrity in genome space. The species exists as a shell of individuals on the edge of the blob of mediocrity. The blob needs to be huge, so that individuals whose genome is one twelfth mutations are still in the blob. Then there can be an equilibrium between back-mutations, taking offspring towards the interior of the blob and other mutations, taking offspring out of the blob and out of the gene pool. This potentially solves the Fermi paradox. Can creatures such as humans actually exist in this universe? It is not enough for natural selection to discover a good genome. Natural selection has to discover a huge blob of mediocrity. Such blobs might be vastly rarer than we realize. This potentially shits on the CRISPR master race. There might be nothing special about the interior of the huge blob of mediocrity.
    - canjobear 9y ago
      What is the "correct genome"? Seems like you could only define it as a local minimum in fitness space, or some kind of attractor.
- SonOfLilit 9y ago
  Related: Hamming's "The Art of Doing Science and Engineering" chapter 9, N-Dimensional Space http://worrydream.com/refs/Hamming-TheArtOfDoingScienceAndEngineering.pdf http://worrydream.com/refs/Hamming-TheArtOfDoingScienceAndEn... (I assume Bret Victor has permission to host the PDF on his website, he is far from an anonymous pirate)
adamnemecek 9y ago
I'm not sure I know what you are talking about but let's not throw away the baby with the bath water.
- kobeya 9y ago
  What he means is that we only really have intuition for 1, 2, and 2.5D visuals, but many areas of mathematics don’t map into low dimensions very well, or do but lose essential properties in the process. Building a low dimensional projection of he problem might prime intuition, but it will also introduce fundamental biases as well. For example, learning geography by flat map projections only. No matter what projection you use there is a trade off, and you end up instilling both the pro and the con of that trade off as intuition.
  - phreeza 9y ago
    Yes, exactly, thanks.
  - adamnemecek 9y ago
    I didn't understand the distributions part.
    - chestervonwinch 9y ago
      It's related to the geometric problems the parent described because probability distributions roughly describe geometric regions (of high probability density) where observations are likely.
    - nitrogen 9y ago
      Part of it may be that in higher dimensions the bulk of a volume is concentrated near the surface. I found https://blogs.msdn.microsoft.com/ericlippert/2005/05/13/high-dimensional-spaces-are-counterintuitive-part-two/ https://blogs.msdn.microsoft.com/ericlippert/2005/05/13/high... with a quick search.
      - 9y ago
zardo 9y ago
That means you need to keep track of what properties actually hold under projection to lower dimensions.
- smallnamespace 9y ago
  Keeping track of properties is not really something 'visualization' necessarily helps with though, more symbolic reasoning through proofs.
  - zardo 9y ago
    I don't mean that spatial reasoning helps with that, I mean that if you do it, you can still apply your spatial reasoning where it's appropriate.
  - dmritard96 9y ago
    I agree it isn't 100% perfect for every situation. But I can think of plenty of instances where ditching colors, lowering resolution, etc. have been totally fine (and often essential) for gaining a level of intuition. As some other comments noted, this intuition may be flawed, mostly because it's hard to know what you don't know. As a result, you are still more informed than you might have been before, but you also might not know much more. however, being able to mix and match properties is really what most good plotting and visualization is all about. having done a bunch of ray tracing, my intuition around lighting and light is much better. I am not even good 'anechdata' so take that for what its worth, but I found visualization to be much more intuitive than reading e&m textbooks/lectures. I'm not sure if knowing a phenomena as bottom up or top down is really a guarantee (nor am I suggesting that symbolic reasoning is bottom up or down), but seeing something is just so efficient for some people. Like anything powerful, it just needs to be used judiciously and with asterisks.
kahoon 9y ago
I think this is a very good point. Years ago people were worried about gradient descent getting stuck at a local minima, plausibly because this problem is very obvious in a 3 dimensional space. In higher dimensions however this problem seems to go away more or less and a lot of worrying about the issue seems to be the result of lower dimensional intuitions wrongly extrapolated to higher dimensions.
- Eliezer 9y ago
  Stuff got stuck in local minima for years before we learned about stuff like momentum and dropout and dropped a ton of GPU power on it.
  - llamaz 9y ago
    When I was implementing a neural network for a university assignment (2 years ago so my memory might fail me), we had to run our algorithm multiple time with different starting positions, then take the minimum of those local minima. I'm not sure what momentum and dropout are, but I agree with Eleizer, without these things (which I didn't use) local minima are a problem.
    - eat_veggies 9y ago
      Dropout is where you randomly remove neurons from your network during training, which prevents them from depending too much on specific neurons (making the output more generalizable). It was developed in 2014 so it would have been brand new tech back when you were in your class.
  - alexcnwy 9y ago
    I think you're misinterpreting the parent who is saying that local minima are not a problem in high dimensions because there is always a dimension to move in that reduces the loss (unlike in lower dimensions where you can get stuck in a point across all dimensions that cannot be locally improved upon)
    - 9y ago
tankenmate 9y ago
Here is a good short document[0] that can help unlock the missing visual intuition of high dimensional data. It does require a lot more thinking than 3d space, but eventually along side the maths it help you to get a "feel" for it. [0] https://www.cs.cmu.edu/~venkatg/teaching/CStheory-infoage/chap1-high-dim-space.pdf https://www.cs.cmu.edu/~venkatg/teaching/CStheory-infoage/ch...

This can also go wrong, for example visualising probability distributions in low dimensions leads to very wrong intuitions about the behavior of high-dimensional dimensional distributions.

vbuwivbiu 9y ago

please elaborate - are you thinking of the curse of dimensionality ?

phreeza 9y ago
There are many examples, one I came across recently is that the large majority of the probability mass of a high-dimensional gaussian distribution is in a shell at a distance from the mean, the mass at the center is actually quite low. Also anything related to topology, which is important when you are looking at decision boundaries, becomes counterintuitive in high dimensions, because so many things can be adjacent at the same time.
- shas3 9y ago
  Theoretically, yes. Can you give a more concrete example? Many hard high dimensional and general topology problems can be visualized through their 2D special cases.
- jstanley 9y ago
  Can you please try and explain why that is? If true, you're very correct that lower-dimensional intuition does not transfer into higher-dimensional spaces: my intuition tells me that a Gaussian distribution drops off as you fall away from the mean, and it's quite easy for me to imagine that in 2 dimensions, 3 dimensions (e.g. by imagining a mound on a plane) and 4 dimensions (e.g. a cloud in 3-space with increased density around the mean). Is my intuition wrong in any of those cases? If so, why? If not, how many dimensions do we need before it becomes wrong?
  - smallnamespace 9y ago
    The center of the distribution always has the highest density, but the ratio of 'probability mass close to centroid' / 'total probability mass' drops off as number of dimensions grows. This is somewhat related to another 'curse of dimensionality' observation, which is that the volume of a hyperball / volume of hyperspace tends towards zero as dimensions grow -- there's just a lot more volume that's in some sense 'far' from the center.
  - orangecat 9y ago
    Because an outlier in any single dimension will put the point outside the "center" of the distribution, and as the number of dimensions increases there's more of a chance of that happening. Say you have an N-dimensional gaussian where each dimension has mean 0 and standard deviation 1. Define the center as the N-dimensional cube whose edges go from -3 to +3 in each dimension. A normally distributed value is within 3 standard deviations of the mean with probability 0.9973, so the probability that an N-dimensional point being in the center is 0.9973^N. With N=4 that's 0.989 which matches your intuition, but at N=1000 it's 0.067 and at N=10000 it's 1.81e-12.
  - nabla9 9y ago
    > it's quite easy for me to imagine that in 2 dimensions It starts to fail really badly when dimension grows. Two simple examples: 1) Consider 3 dimensional unit sphere centered at origin and unit cube centered at origin. Cube is clearly completely inside the sphere. Now generalize to n-dimensions. Hyperdimensional volume of hypercube with side length 1 moves almost completely outside the n-sphere with radius 1 when n-grows. 2) Alternatively almost all volume of n-sphere is close to the surface. These are all very counterintuitive, yet simple to check toy examples. When you start to integrate over more complex multidimensional function, things get weird really fast.
    - tzahola 9y ago
      >Alternatively almost all volume of n-sphere is close to the surface. How does this go against intuition? Intuition from 1/2/3d tells me that the volume of an N-ball is O(r^N), and indeed it is the case in higher dimensions. Therefore it’s easy to see that the difference between the volume of an N-ball of radius r and an N-ball of radius (r + epsilon) will grow exponentially with N.
- sevenfive 9y ago
  Yeah, but the distance from the center is itself just a one-dimensional gaussian...
- alan-crowe 9y ago
  This is the problem that makes my brain melt when I try to think about genetics and mutational load. The naive idea is that a species, S, has a correct genome, G, but mutations build up, increasing with each generation. Presumably mutations build up until they are common enough that back-mutations are a thing. Then there is an equilibrium. In a fecund species, each individual has many children, but most have a higher mutational load, many have the save mutational load, and a lucky few have a smaller mutational load, closer to G, the correct genome. Differential reproductive success then maintains the equilibrium. I don't see how the numbers are supposed to work out for large mammals, with each female having under a dozen offspring. To have a decent chance of a back-mutation, the typical member of the species would need one twelfth of their genome to be deleterious mutations. Meanwhile, people are thinking about using CRISPR to correct the human genome, creating unusually happy, healthy people. The underlying thought is that the correct genome is best. But why do we think that the correct genome works at all? Most of the population is in a shell at a distance from the correct genome, the number at the center is actually quite low. Given the combinatorics, with two to the millions of possible genomes, but populations in the millions, the number at the center, or even close, is actually zero. Maybe the correct genome codes for a sickly, miserable individual? My current guess is that the evolution of large mammals with few offspring is constrained by genetic load considerations. It is not sufficient, (or even necessary) for the correct genome to be any good. There needs to be a big blob of mediocrity in genome space. The species exists as a shell of individuals on the edge of the blob of mediocrity. The blob needs to be huge, so that individuals whose genome is one twelfth mutations are still in the blob. Then there can be an equilibrium between back-mutations, taking offspring towards the interior of the blob and other mutations, taking offspring out of the blob and out of the gene pool. This potentially solves the Fermi paradox. Can creatures such as humans actually exist in this universe? It is not enough for natural selection to discover a good genome. Natural selection has to discover a huge blob of mediocrity. Such blobs might be vastly rarer than we realize. This potentially shits on the CRISPR master race. There might be nothing special about the interior of the huge blob of mediocrity.
  - canjobear 9y ago
    What is the "correct genome"? Seems like you could only define it as a local minimum in fitness space, or some kind of attractor.
SonOfLilit 9y ago
Related: Hamming's "The Art of Doing Science and Engineering" chapter 9, N-Dimensional Space http://worrydream.com/refs/Hamming-TheArtOfDoingScienceAndEngineering.pdf http://worrydream.com/refs/Hamming-TheArtOfDoingScienceAndEn... (I assume Bret Victor has permission to host the PDF on his website, he is far from an anonymous pirate)

adamnemecek 9y ago

I'm not sure I know what you are talking about but let's not throw away the baby with the bath water.

kobeya 9y ago
What he means is that we only really have intuition for 1, 2, and 2.5D visuals, but many areas of mathematics don’t map into low dimensions very well, or do but lose essential properties in the process. Building a low dimensional projection of he problem might prime intuition, but it will also introduce fundamental biases as well. For example, learning geography by flat map projections only. No matter what projection you use there is a trade off, and you end up instilling both the pro and the con of that trade off as intuition.
- phreeza 9y ago
  Yes, exactly, thanks.
- adamnemecek 9y ago
  I didn't understand the distributions part.
  - chestervonwinch 9y ago
    It's related to the geometric problems the parent described because probability distributions roughly describe geometric regions (of high probability density) where observations are likely.
  - nitrogen 9y ago
    Part of it may be that in higher dimensions the bulk of a volume is concentrated near the surface. I found https://blogs.msdn.microsoft.com/ericlippert/2005/05/13/high-dimensional-spaces-are-counterintuitive-part-two/ https://blogs.msdn.microsoft.com/ericlippert/2005/05/13/high... with a quick search.
    - 9y ago

zardo 9y ago

That means you need to keep track of what properties actually hold under projection to lower dimensions.

smallnamespace 9y ago
Keeping track of properties is not really something 'visualization' necessarily helps with though, more symbolic reasoning through proofs.
- zardo 9y ago
  I don't mean that spatial reasoning helps with that, I mean that if you do it, you can still apply your spatial reasoning where it's appropriate.
- dmritard96 9y ago
  I agree it isn't 100% perfect for every situation. But I can think of plenty of instances where ditching colors, lowering resolution, etc. have been totally fine (and often essential) for gaining a level of intuition. As some other comments noted, this intuition may be flawed, mostly because it's hard to know what you don't know. As a result, you are still more informed than you might have been before, but you also might not know much more. however, being able to mix and match properties is really what most good plotting and visualization is all about. having done a bunch of ray tracing, my intuition around lighting and light is much better. I am not even good 'anechdata' so take that for what its worth, but I found visualization to be much more intuitive than reading e&m textbooks/lectures. I'm not sure if knowing a phenomena as bottom up or top down is really a guarantee (nor am I suggesting that symbolic reasoning is bottom up or down), but seeing something is just so efficient for some people. Like anything powerful, it just needs to be used judiciously and with asterisks.

kahoon 9y ago

I think this is a very good point. Years ago people were worried about gradient descent getting stuck at a local minima, plausibly because this problem is very obvious in a 3 dimensional space. In higher dimensions however this problem seems to go away more or less and a lot of worrying about the issue seems to be the result of lower dimensional intuitions wrongly extrapolated to higher dimensions.

Eliezer 9y ago
Stuff got stuck in local minima for years before we learned about stuff like momentum and dropout and dropped a ton of GPU power on it.
- llamaz 9y ago
  When I was implementing a neural network for a university assignment (2 years ago so my memory might fail me), we had to run our algorithm multiple time with different starting positions, then take the minimum of those local minima. I'm not sure what momentum and dropout are, but I agree with Eleizer, without these things (which I didn't use) local minima are a problem.
  - eat_veggies 9y ago
    Dropout is where you randomly remove neurons from your network during training, which prevents them from depending too much on specific neurons (making the output more generalizable). It was developed in 2014 so it would have been brand new tech back when you were in your class.
- alexcnwy 9y ago
  I think you're misinterpreting the parent who is saying that local minima are not a problem in high dimensions because there is always a dimension to move in that reduces the loss (unlike in lower dimensions where you can get stuck in a point across all dimensions that cannot be locally improved upon)
  - 9y ago

tankenmate 9y ago

Here is a good short document[0] that can help unlock the missing visual intuition of high dimensional data. It does require a lot more thinking than 3d space, but eventually along side the maths it help you to get a "feel" for it. [0] https://www.cs.cmu.edu/~venkatg/teaching/CStheory-infoage/chap1-high-dim-space.pdf https://www.cs.cmu.edu/~venkatg/teaching/CStheory-infoage/ch...