4 ms·
Another counterintuitive result in this vein is to think about randomly sampling uniformly distributed points in a sphere. As the dimension rises, the chance of
by tel 10y ago
Another counterintuitive result in this vein is to think about randomly sampling uniformly distributed points in a sphere. As the dimension rises, the chance of finding a sample within, say, r/100 of the surface goes to infinity. In high dimensions, all of the mass of the sphere is concentrated at the edge.
- falcor84 10y agoSorry for the nitpick, but I assume you meant that the chance goes to 100%, rather than infinity.
- tel 10y agoYup! But too late to edit :( Thanks!
- wyager 10y agoThat makes sense now that you mention it. The equation for the surface area of an n-sphere goes up as r^n, and the volume growth at any given radius is equal to the surface area at that radius.
- wbl 10y agoI've heard this phrased as "oranges are not a fun fruit in high dimensions".
- dllthomas 10y agoPithy.
- pakl 10y agoExactly! Now consider that deep networks that classify images are tasked with getting reliable statistics in very high dimensional spaces. Conv nets are being forced to map from high dimensional spaces down to a very low dimensional categorical decision. This is why weird classification errors you see in "adversarial examples" keep popping up. The learned classification boundaries are very spiky, and it's easy for the world to fall in between the spikes. More data can't solve it (at least not practically) because there are way too many gaps between the spikes. A more tractable approach is to learn to use dynamics for perception rather than ("just") statistics. The dynamical physics of a ball rolling is much simpler (lower dimensional, more tractable) than a statistical view of millions of differently illuminated pixels hitting a camera. (A colleague of mine has a blog post on this issue of "statistics and dynamics" at http://blog.piekniewski.info/2016/11/01/statistics-and-dynamics/ http://blog.piekniewski.info/2016/11/01/statistics-and-dynam...)
- js8 10y agoYour point about dynamics kinda reminds me of Chomsky's critique of statistical approaches to AI, for example here: http://norvig.com/chomsky.html http://norvig.com/chomsky.html
- pakl 10y agoYes, but in direct contrast to Chomsky (who would say there's not enough data/time for kids to learn from) I am saying that there is a ton of rich dynamical data in the world around us all the time. Plenty to learn from. Just plug a webcam into an adequate system and allow it to learn dynamics by trying to predict what it will see next. Chomsky is almost right for the wrong problem: there won't ever be enough human-labeled data for good generalization. ;)
- js8 10y agoI think what Chomsky was saying in that particular debate was that purely statistical methods do not lead to true understanding, as opposed to more phenomenological theory.
- theoh 10y agoI've heard this phrased as "objects in high dimensions are mostly surface", which makes clear that it's not that there's a less dense "core" but that every interior point is near the surface.
- Retric 10y agoThat seems very intuitive if you think about the difference between a line segment and a circle. In 1d space a segment that extended S1: from 0 to 1 covers half the space of one that extends from S2: 0 to 2. In x,y space you have that same 50:50 ratio at y=0. But, for every y > 0 the ratio decreases as a higher percentage of S1 is used up just reaching that y height than S2, until all of it is used up and the top segments are 100% s2. In x,y,z space you get that same picture at z=0. But at any z > 0 more of that S1 is used to than the S2 segment so the ratio must be lower than in x,y or x space. Moving to 4D and up the same simple idea just continues.