Very cool. My only niggle is that PCA doesn't 'feel like' Kolmogorov Complexity to me: it's much closer to Shannon information than Algorithmic information. Sh

Very cool. My only niggle is that PCA doesn't 'feel like' Kolmogorov Complexity to me: it's much closer to Shannon information than Algorithmic information. Shannon information depends on population statistics of messages, usually has a 'shallow' and domain-specific interpreter (to turn encodings back into messages, e.g. like a Huffman tree), and tries to minimise the average length of a set of encoded messages. In this case the NN is the interpreter, the length of the encoding is the number of deformation axes, and the population is the training data (which we hope is representative of the real data). Algorithmic information is different. It depends on patterns within an individual message, rather than across a population. The interpreter is a Universal Turing Machine (or equivalent), which is general-purpose. Encodings are programs which output a message, and can be arbitrarily 'deep' by running for an arbitrary amount of time. Another way to look at it is: using PCA to solve the problem of 'automatically decide which deformation axes to include' is slightly interesting; much more interesting is the problem of 'how can we automatically decide that this data should be modelled as a set of deformations along various axes?'. That's a much harder problem, and tends to require a lot of search through program-space (whether NNs or GOFAI).