3 ms·
To summarize: + An autoencoder is a stabilizer of the input f: it maps it to itself. + Imagine that the space of autoencoders forms a group. + Learning stops
by MrQuincle 10y ago
To summarize:
+ An autoencoder is a stabilizer of the input f: it maps it to itself.
+ Imagine that the space of autoencoders forms a group.
+ Learning stops as soon as a stabilizer is found.
+ If the search is a Markov chain, the bigger the stabilizer the sooner it will be hit.
+ The group structure implies that this big stabilizer corresponds to a small orbit.
+ The orbit of an element x in X is the set of elements x can be moved to by the elements of G (the group).
+ The stabilizer is the set of group actions that map x to itself.
+ Blog post explaining fundamentals: https://gowers.wordpress.com/2011/11/09/group-actions-ii-the-orbit-stabilizer-theorem/ https://gowers.wordpress.com/2011/11/09/group-actions-ii-the...
+ Reconstruction in deep NNs is often guided by an l2 distance. If there are competing feature sets, gradient descent moves the configuration to one of the stabilizers.
- Note. Is this indeed the case? Is there a WTA mechanism at play? Is this preferable?
- Note. In this view a NN is a MCMC that is stopped prematurely.
+ The probability that a network "discovers" a stabilizer for the signals f_i depends on the volume of the stabilizer.
- Would we be able to construct a PDF by running an MCMC till we are in a high probability region? Or does this only work if we're interested in modes?
+ A neural network operation is not a group action but can be seen as a "shadow group" in some auxiliary space (theorem 4.1, 4.2).
+ The size of the stabilizers is preserved in the original space (theorem 4.3).
+ Next levels exhibit again symmetry, generalized edges lead to figures like trapezoids, triangles, and butterflies.
Most important take-away: representations that exhibit a lot of symmetries are the ones that are easiest/fastest to find.
- Ericson2314 10y agoThis could be the least dismissive and anti-intellectual tl;dr I've seen here.
- deleted 10y ago[deleted]
- kobeya 10y agoYou are my hero.
- huac 10y ago> + Reconstruction in deep NNs is often guided by an l2 distance. If there are competing feature sets, gradient descent moves the configuration to one of the stabilizers. > - Note. Is this indeed the case? Is there a WTA mechanism at play? Is this preferable? > - Note. In this view a NN is a MCMC that is stopped prematurely. Generally the 'performance' of the autoencoder is measured at the reconstruction step, i.e. what is the reconstruction accuracy of the autoencoder? This is an L2 loss function.
- Ar-Curunir 10y agoThis is pretty cool; the link between machine learning and group theory is neat. Do you know if this link is new? I've not heard of it before. Thanks for the summary!
- MrQuincle 10y agoThis particular link seems new indeed. There have been other links listed on HN in the past. One that came back a few times is on renormalization group theory (actually a semigroup). Currently my new angle of attack is to check what people have to say who come from a representation theory angle. I also liked this paper by Persi Diaconis a lot: http://statweb.stanford.edu/~cgates/PERSI/papers/The%20MCMC%20Revolution.pdf http://statweb.stanford.edu/~cgates/PERSI/papers/The%20MCMC%... I also like that on his wikipedia page (https://www.wikiwand.com/en/Persi_Diaconis https://www.wikiwand.com/en/Persi_Diaconis) it's stated that: According to Martin Gardner, at school, Diaconis supported himself by playing poker on ships between New York and South America. Gardner recalls that Diaconis had "fantastic second deal and bottom deal". :-)
- jbpetersen 10y agoI can't possibly be the only one here who wishes articles were presented in this format, with each of the points clickable for more details, such that the whole thing is a tree structure of summarized points.