12 ms·
An idea from physics helps AI see in higher dimensions
- throwaway_tech 7y agoWow this is actually fairly close to my prediction for the HN Next Decade Prediction post. Edit: I should say it is a big step in the direction of my prediction.
- gojomo 7y agoWhat prediction, what post?
- throwaway_tech 7y agoPost: Ask HN: A New Decade. Any Predictions? https://news.ycombinator.com/item?id=21941278 https://news.ycombinator.com/item?id=21941278
- rckoepke 7y agoyeah, which comment?
- pseudosudoer 7y agoI think OP realized that linking the comment will reveal their real HN account name lol.
- throwaway_tech 7y agoNope, I just didn't realize linking to the post wasn't enough to find my comment. Figured once there it would be pretty trivial to search for my name. anyway linked above you, so know you have something else to lol about
- pseudosudoer 7y agoWith "throwaway" as the prefix to your account name, it should be fair to assume that's not your main account... Also, I don't really see how your prediction has anything to do with gauge theory applied to CNN, unless I'm missing something much deeper here?
- throwaway_tech 7y agohttps://news.ycombinator.com/reply?id=21945829&goto=threads%3Fnext%3D21948092%26id%3Dthrowaway_tech%2321945829 https://news.ycombinator.com/reply?id=21945829&goto=threads%... >My big prediction is in physics: Thanks to Einstein we live in a 4D reality, 3 spatial dimensions and the dimension of time. In the next 10 years I predict our understanding of physics will evolve (with a confirmation through observation) from us being 3D beings living in a 4D reality to...something more.
- sillysaurusx 7y agoOne of the strangest things in AI, to me, is that you can average the weights of multiple different models to create a single model that’s better than any of the individuals. This is how distributed training often works, for example. Data parallelism. I don’t understand why it still works in higher dimensions, but it seems to. The intuition is that the multiple models are “spinning” around the true solution, so averaging gives the final result more quickly. But it works even early in the training process.
- im3w1l 7y agoI believe it works in distributed training because the models never have time to diverge far enough to be "incompatible".
- deleted 7y ago[deleted]
- joker3 7y agoConcentration of measure. If you have a quantity that depends on a large number of random variables but not too strongly on any small subset of them, it tends to behave like a constant. That's the intuition behind the law of large numbers, the central limit theorem, a bunch of concentration inequalities, and model averaging.
- marcosdumay 7y agoAnd, of course, there is nothing that says this would work for intermediate layers, since the sample dimensions may get there from any input. What works is averaging similar networks and averaging your networks a lot of times.
- rprenger 7y agoWhen we use data parallelism, we're summing/averaging the gradients induced by different parts of the data, not the weights of the model itself. When using multiple models for ensemble methods, we're summing/averaging the output of the models for a sample. We're not summing/averaging the weights of the models themselves. While averaging the weights of several models might work on a given problem, it definitely doesn't work in general. w2tanh(w1x) = -1w2tanh(-w1*x) But if you average the weights in those two equivalent models you get 0. If you're talking about asynchronous data parallelism, then there can be some averaging of weights, but they all start with the same weights and are re-synched often enough that weights are never too different to break it.
- BubRoss 7y agoThis article is all over the place with almost no substance. It starts talking about Einstein's theory of relativity, it says this definitively solves curved data and gives almost no insight into what is actually different. It is so bad it makes me want to avoid this site all together.
- gojomo 7y agoThe article includes multiple links to the related underlying research papers, for people who need more substance.
- BubRoss 7y agoMy point is that the article is worse than just having a link to the paper. It just says it might lead to cures for diseases, better self driving cars and lots of other abstract sensational nonsense. It could be used as a template and any new machine learning paper could just be linked to it.
- eli_gottlieb 7y agoIf you think gauge invariance on Riemannian manifolds is an empty topic with no substance, you might not want to be working in machine learning.
- BubRoss 7y agoDid you read my comment before writing this?
- empath75 7y agoPretty amazing work -- a couple of thoughts: 1) The article doesn't say this, but dimensions don't always have to do with locations in space and time, you can treat any value that can continuously vary as a dimension -- for example, a person might have dimensions for personality type, age, hair color, etc, etc.. Seems like they could use this technique to better train CNNs to recognize patterns in a lot of data besides imagery -- fraud detection based on credit card transactions, for example. 2) There are a lot of local and global symmetries in physics -- i wonder what new capabilities adding them to a CNN would enable?
- mywittyname 7y agoThe article is talking specifically about performing convolutions on higher dimensional manifolds. This is different from the more broad concept of data dimensionality typically associated with AI/ML. Without repeating the article too much, this is important because it can be used to learn very complex systems from a series of lower dimensional projections. Such as, creating a 3d map of a dog from a collection of 2d images of dogs. The resulting system can better detect a dog in a position it's never seen because the CNN has built into it the relationship between 3d space and the 2d representation of that space.
- empath75 7y agoThis is a serious question because I don’t know: what’s the difference between high dimensional data and high dimensional manifolds?
- hansvm 7y agoThis isn't a complete answer, but consider a series of data points describing a line. E.g., (0,0,0,0), (1,1,1,1), (2,2,2,2), .... It's possible to have lines even in any n-dimensional space, but the data is "high dimensional" -- you're using a large number of variables to describe a fairly simple process. Contrast that example with any sufficiently large, uniform sampling of the n-dimensional unit cube. The data is inherently high dimensional, and any attempt (attempts from a certain class of allowed methods -- no need to burden ourselves with the details) to reduce from n to k<n dimensions will throw away some important information about the structure of the data. Interestingly, it's very possible to have a low dimensional representation of a high dimensional process. One of the other comments mentioned the example of a photograph representing a 3D scene (also containing a week, implicit view into some other variables like temperature). That transformation from 3D->2D is inherently lossy, but for some kinds of problems a finite sampling of a low dimensional representation allows you to uniquely reconstruct a high dimensional data representation. I haven't read the article yet, but the other comments seem to indicate something of that flavor happening here.
- MrQuincle 7y agoWhat happens if there are multiple 3D or 4D objects? Do we then need an attention mechanism as well? Or is there some topology where a "where vs what" pathway emerges naturally?
- starpilot 7y agohis name is taco? really?
- deleted 7y ago[deleted]
- hanniabu 7y agoIs multidimensional AI the new 2020 buzzword?
- z3c0 7y agoI sure hope so. Telling people I build multidimensional data structures for a living has only yielded glazed-over eyes thus far.
- nixpulvis 7y agoGlazed over in awe or boredom?
- andai 7y agoLike those little question marks that would appear over AI that was "thinking" in 90's games.
- 77544cec 7y ago- I'm a developer. - Huh ? - I build websites types on an air keyboard.
- catalogia 7y ago"Developer" is a word that requires some context. A stranger might not know if you worked in construction or on computers.
- toper-centage 7y agoOn a professional context I'll say I'm a software developer. If I want to brag a bit I'll say I'm a software engineer. If I'm with friends I'll say I'm a programmer. If I'm with family or older people I'll dip my toes with "I work with computers" and maybe further explain if prompted.
- 7y ago
- beefield 7y ago_I_ would like to have a VR experience in higher dimensions. It should not be completely impossible to build some kind of actuators that I can somehow attach to my body to sense my orientation and acceleration in fourth dimension.
- uj8efdkjfdshf 7y agoMay I recommend 4D Toys? It's made by the same guy who's developing Miegakure and it has a VR version. The controls are a bit limited though in that user-initiated rotations are limited to 3D.
- beefield 7y agoI thought that is me being stuck in 3d and the toys around me moving in 4d? As I would like to move myself in 4d...
- 6nf 7y agoThere's a slider to move yourself in the extra dimension.
- _wzsf 7y agoI continue to be unceasingly disappointed by his failure to release Miegakure after all these years.
- gambler 7y agoCan someone explain to me why advances in actual model performance come from using analogies from physics when there are papers that supposedly provide a mathematical explanation of convolution? "A Mathematical Theory of Deep ConvolutionalNeural Networks for Feature Extraction": https://arxiv.org/pdf/1512.06293.pdf https://arxiv.org/pdf/1512.06293.pdf "Understanding Convolutional Neural Networks with A Mathematical Model": https://arxiv.org/pdf/1609.04112.pdf https://arxiv.org/pdf/1609.04112.pdf
- conjectures 7y agoBecause it's not a standard convolutional net by the description. The difference is: A) Studying an existing technique with math. B) Coming up with a new technique. You could get a modern engineering consultancy to review your steam engine, but it would still be a steam engine.
- mojomark 7y agoRelated G-CNN video: https://youtu.be/wZWn7Hm8osA https://youtu.be/wZWn7Hm8osA
- etaioinshrdlu 7y agoThis is super cool and I'm pretty sure this is basically topology. The article was pretty hard to read though. It reminds me a little bit of the https://en.m.wikipedia.org/wiki/Hairy_ball_theorem https://en.m.wikipedia.org/wiki/Hairy_ball_theorem
- improbable22 7y agoNot really topology, it's more like group theory, and representations. Ordinary convnets are a way of building in translational symmetry, which is the group R^2 (in the plane). The work being described extends this to larger symmetry groups, such as rotations of a molecule in 3D (which is SO(3)). For either of these, you can work in Fourier space instead of real space, where convolutions become products. For ordinary convnets means ordinary FFT, but nobody does that as translating to neighbouring pixels is simple enough. Rotations aren't so simple, and so working in Fourier space can be an efficient way to do things. And the connection to physics is really just that the representation theory of SO(3) is a bread-and-butter exercise there, the basis of atomic theory.
- jhisiow9839 7y agoIs this very different from a graph convolutional network (GCN)? Seems like a GCN would have a lot of the same equivariabne properties (i.e. orientation, units of measure, etc)?
- cliqueiq 7y agoI thought quantamagazine was above publishing click-bait, but I guess not. Neural networks already see in "higher dimensions" (whatever that means). Anyone who's ever used neural networks already knows each neuron's branch (i.e. dendrite) of an N-sized vector can already be though of as a "dimension" of a data set. CNN (convolutions) flatten that data (reduce it or seeing the same pattern over less "dendrites", much like PCA, etc.). CNNs only make sense when working with image data anyways.
- ipsum2 7y ago> CNNs only make sense when working with image data anyways. Not true, CNNs are used for audio and text as well. I don't think the title is clickbait, you may be misinterpreting it. Its referring to using CNNs on higher dimensional inputs, not that the layer has multiple dimensions (which has been done since the creation of convnets)
- jhj 7y ago> CNNs only make sense when working with image data anyways Not true, N-dimensional convnets, 1-d convnets (for NLP and time series analysis), spatially sparse convnets, graph and non-Euclidean space convnets, ... exist and are used. CNNs are akin to multiscale wavelet transforms. They can be applied on different spaces (just as graph wavelet transforms exist).
- peter_d_sherman 7y agoExcerpt: "Now, researchers have delivered, with a new theoretical framework for building neural networks that can learn patterns on any kind of geometric surface. These “gauge-equivariant convolutional neural networks,” or gauge CNNs, developed at the University of Amsterdam and Qualcomm AI Research by Taco Cohen, Maurice Weiler, Berkay Kicanaoglu and Max Welling, can detect patterns not only in 2D arrays of pixels, but also on spheres and asymmetrically curved objects. “This framework is a fairly definitive answer to this problem of deep learning on curved surfaces,” Welling said."
- ganzuul 7y agoSomething that could go by the same title is the use of tensor networks for ML. I think it works like a pre-optimization step by dimension reduction of the solution space, but if someone could give the right intuitive explanation I'd be much obliged. It seems to be a way to lessen inductive bias by making decisions about available ML algos. That is, it vastly increases the solution space but remains effective by omitting unlikely solutions.
- irenedowd 7y agoHi. I ran over a generally excellent Hacker Goatse Security. They have assisted with a ton of issues like Phone Hack, Account Hack, Clear Debts, Grade overhaul E.t.c Contact: goatsesecure (at) gmail Text/Whatsapp: +1 646 389 4585
- cosmic_ape 7y agoHype pipeline: 1) Take a banal feature engineering work. 2) Add Albert Einstein reference. 3) Profit.
- openasocket 7y agoThis is a really interesting application of differential geometry in machine learning! And the allusion at the end to having the system eventually learn the symmetries of the system and make sure of that is really interesting. All the examples they gave were very physical, like climate models, but I imagine you could find symmetries in much more abstract problems that may not be intuitive.