Over the past few months, I've been learning group theory, representation theory, and a little bit about reproducing kernel Hilbert spaces, and I think these co

Over the past few months, I've been learning group theory, representation theory, and a little bit about reproducing kernel Hilbert spaces, and I think these concepts also tie in with the author's vision for the future of neural networks. We are often interested in studying a set of objects in a way that is invariant to some type of transformation. Essentially, we want to retain all of the information describing an object except for that which is necessary to distinguish the object from all transformed versions of itself. As a motivating example, let's say we want to compare a collection of photographs. One difficulty is that each photograph may have an arbitrary/unknown rotation applied to it. How, then, can we create a representation of each photo that does not depend on its rotation? The trivial solution is to map each photo to the element of a singleton. This solves the rotation problem but unfortunately removes all rotationally invariant information from the photos as well. So our goal is to find the boundary that splits the information describing each photo into "invariant" and "non-invariant" partitions. One way to do this is to train a neural network on arbitrarily rotated versions of many photos. The network will then evolve to extract the unnecessary information from each photo (compressing them in a sense). The downside of using a neural network is that the training process can take a heck of a long time to figure out the invariants, and it isn't even guaranteed to work that well. And that's just for the preprocessing step; the rest of the network will take forever to train. So what can we do? Another idea is to use the Gram matrix (a.k.a. the kernel matrix) corresponding to a set of n points in R^2. The Gram matrix preserves all non-rotational information about the set of points; however, it is also overcomplete, meaning that we started with 2n real numbers, and now we have n^2 real numbers (or n(n-1)/2 if we consider just the upper triangular portion of the Gram matrix). This is less than ideal. Representation theory furnishes a better solution to the problem by providing a means of identifying the fundamental degrees of freedom that characterize a set of transformations (i.e., a group). A representation ρ of a group G is a homomorphism from G to the general linear group on a vector space V. A representation is considered irreducible if there exists no nontrivial proper subspace W ⊂ V for which ρ(g)(w) ∈ W for all w ∈ W, g ∈ G. So assuming you can find an irreducible representation, the remaining task is to figure out how to label (index) the mappings. A familiar example of a set of labeled mappings of an irreducible representation is the set of spherical harmonics: these functions constitute an irreducible representation of the group SO(3) (rotations in 3D space). The functions are indexed by integers l ≥ 0, |m| ≤ l. Every equivalence class {f(x, y, z) ~ (Qf)(x, y, z) | x, y, z ∈ R, Q ∈ SO(3)} maps to a unique linear combination of the spherical harmonics (augmented with a set of radial functions). Thus, the coefficients of these basis functions contain only the rotationally invariant information describing a function over R^3, and nothing more (these are what you want to feed into your neural network!). In a similar vein, the symmetric group Sn (the group of all permutations of n objects) has irreducible representations that are indexed by the Young tableaux (unique geometric shapes corresponding to integer partitions). So you can isolate the permutationally invariant information that specifies a function as well.