Consider an analog. Say you wanted to design an image compression format in the spirit of jpeg, so you take a bunch of 8x8 blocks of image data and you use a p

Consider an analog. Say you wanted to design an image compression format in the spirit of jpeg, so you take a bunch of 8x8 blocks of image data and you use a principal component analysis to design an optimal orthogonal transformation to apply to 8x8 blocks to compact their energy into fewer coefficients so you can compress it. But collecting lots of image data is hard so someone comes along and points out that the most important characteristic of your images is that they're appear drawn from an autoregressive process... That is that each pixel is (say) 95% correlated to the pixel 1 away, 95%^2 to the pixel two away, 95^3 three away and so on. So to compute the PCA you only need the covariance matrix anyways, so you just generate an AR0.95 covariance matrix and use the transform derived from that. And you find this works pretty well too. This work is along those likes, they're generating random images with some simple natural-like statistical properties and training the first part of the classifier using them and getting useful results. One interesting promising part is that this line of thinking may result in better network designs or insights that allow skipping training these initial layers: Going back to the block transform example, the next step would be to notice that the PCA of the AR0.95 matrix is the discrete cosign transformation, which is agreeable to an extremely efficient implementation.