5 ms·
A good way to conceptualize what’s going on is not the idea that balls become "spiky" in high dimensions – like the article says, balls are always perfectly sym
by Sharlin 2y ago
A good way to conceptualize what’s going on is not the idea that balls become "spiky" in high dimensions – like the article says, balls are always perfectly symmetrical by definition. But it’s the box becoming spiky, "caltrop-shaped", its vertices reaching farther and farther out from the origin as the square root of dimension, while the centers of its sides remain at exactly +-1. And the 2^N surrounding balls are also getting farther from the origin, while their radius remains 1/2. Now it should be quite easy to imagine how the center ball gets more and more room until it grows out of the spiky box.
- pfortuny 2y agoExactly: a corner of a square covers 1/4 of that part of the plane. A corner of a cube covers 1/8, a corner of a hypercube in dimension n covers just 1/(2^n) of the space. But each side/face/hyperface divides the plane/space/n-dim space just in half.
- beretguy 2y agoOk, now i started to understand something. Thank you.
- jheriko 2y agoeven then the edges do not suddenly curve. its just all round a bad analogy.
- petters 2y agoVery good to see this as the top-voted comment. I completely agree that this seems like a more natural explanation of what is going on.
- ColinWright 2y agoBut there is another way to think of the high-dimensional balls where "spikey" is the right visualisation. Consider the volume of a cap. Take a plane that is 90% of the distance from the centre to the edge, and look at what percentage of the volume is "outside" that plane. When the dimension is high, that volume is negligible. And when the dimension is really high you can get quite close to the centre, and still the volume you cut off is very small. In our 3D world the closest thing that has this property is a spike. You can cut off quite close to the centre, and the volume excised is small. The sense in which a high-dimensional ball is not a spikey thing is in the symmetry, and the smoothness. So when you want to develop an intuition for a high dimensional ball you need to think of it as simultaneously symmetrical, smooth, and spikey. Then think of another five impossible things, and you can have breakfast.
- Hugsun 2y agoThat's a good point. High dimensional objects can obtain very unintuitive properties, like you describe. This to me feels similar in many ways to how a corner in a high dimensional n-cube, although 90 degrees, no matter how you measure it, seems extremely spiky. As the shape does not increase in width, but the corners extend arbitrarily far away from the center. A property reserved for spiky things in 3D.
- dawnofdusk 2y agoI think our geometric intuition could never be good for a high dimensional object. Consider a sphere in 3D. It's represented by the points which satisfy x^2 + y^2 + z^2 = 1. Because there are only three coordinates, knowing the value of one of them greatly reduces the possibilities for the other two. For example, if I know z is close to 1, then my point is close to the north pole (x and y are both close to zero). However, if I have the n-sphere x_1^2 + x_2^2 + ... + x_n^2 = 1, knowing the value of x_1 gives me very little information about all the other coordinates. And humans' interaction with the geometry is reality is somewhat limited to manipulating one coordinate at a time, i.e., our intuition is for built on things like moving our body linearly through space, not dilating the volume or surface area of our bodies.
- saagarjha 2y agoIf x_1 is close to one, then the more x_n values there are the closer they are to zero.
- blaufuchs 2y agoTFA does say this too, right after the Hamming lecture. "So instead of considering n-balls to be spiky, it’s the space around them that outgrows them."
- Hugsun 2y agoTo be clear: the commenter you replied to just seems to be reiterating the idea, so I'm not accusing them of not reading the article. You should have seen how few replies read the last article I posted. https://news.ycombinator.com/item?id=40525629 https://news.ycombinator.com/item?id=40525629 The majority of the comments, including all the top ones, expressed insights as original, that were pretty thoroughly analyzed in the article. Just read my mildly frustrated replies.
- Sharlin 2y ago"Space is growing around the ball" just by itself doesn’t give much of an intuition at least to me, so I just wanted to elaborate on a way to think about it.
- Hugsun 2y agoYep, thanks for the great contribution!
- gcanyon 2y agoI read the article and the threads (to a point). It's hilarious that in the article you describe LLMs as being like hasty people jumping to conclusions, and the commenters on HN do exactly that thing in their comments.
- Hugsun 2y agoHaha, that's a really funny similarity. I appreciate you sharing it :)
- ColinWright 2y ago> The majority of the comments, including all the top ones, expressed insights as original, that were pretty thoroughly analyzed in the article. Just read my mildly frustrated replies. I have thoughts about this, but I can find no way to contact you to open a conversation. You might want to think about adding something to your HN profile. Meanwhile ... from this and other sources, I feel your pain.
- pfortuny 2y agoIn some sense, a ball is more “natural” than a cube in Euclidean n-space. Once you add the metric, cubes become artificial constructs (despite being the natural elements in just product spaces).