I said above that my guess is it may be making some small gains by learning "pixels are 20% more likely to be black than white" or something like that. Which fi

I said above that my guess is it may be making some small gains by learning "pixels are 20% more likely to be black than white" or something like that. Which fits with the image he provided. There may be regularities in deep NNs. For example, many learned features are basically exactly the same but rotated or translated in a different way. This is because NN's aren't inherently invariant to rotations or translations, so they have to learn multiple features for every possible variation. Another possible regularity is that the weights tend to cluster on specific sections of the input image and ignore the rest. But this entirely depends how the weights are projected into an image. The image provided looks like random static. The standard method is to just create a separate image for every feature, and for every pixel becomes the weights of the NN from that input pixel. Like this: http://fastml.com/images/deep-learning-made-easy/layer2visualization.png http://fastml.com/images/deep-learning-made-easy/layer2visua... Standard image compression software should be able to work with that. But this won't work in higher layers, because there are only 3 to 4 color channels in an image.