3 ms·

Yes, it is pretty noisy. My assumption is that PNG is pretty good at encoding 2D patterns too, and there are some patterns: https://github.com/liuliu/klaus/blob

by liuliu 12y ago

Yes, it is pretty noisy. My assumption is that PNG is pretty good at encoding 2D patterns too, and there are some patterns: https://github.com/liuliu/klaus/blob/master/ccv/ccvResources/layer-11.png https://github.com/liuliu/klaus/blob/master/ccv/ccvResources... This is a big hack though. I am pretty sure there are better dictionary-based compression method at the last step, but pngcrush gives me some nice small gain already :)

Houshalter 12y ago
It's literally indistinguishable from random static: http://silverspaceship.com/static/shot_2.png http://silverspaceship.com/static/shot_2.png My guess is it may be making some small gains by learning "pixels are 20% more likely to be black than white" or something like that. Dictionary wouldn't work either, unless you think the NN is more likely than random to produce sequences of bits that are exactly the same. There may be regularities in deep NNs. For example, many learned features are basically exactly the same but rotated or translated in a different way. This is because NN's aren't inherently invariant to rotations or translations, so they have to learn multiple features for every possible variation. Another possible regularity is that the weights tend to cluster on specific sections of the input image and ignore the rest. But this entirely depends how the weights are projected into an image. The image provided looks like random static. The standard method is to just create a separate image for every feature, and for every pixel becomes the weights of the NN from that input pixel. Like this: http://fastml.com/images/deep-learning-made-easy/layer2visualization.png http://fastml.com/images/deep-learning-made-easy/layer2visua... Standard image compression software should be able to work with that. But this won't work in higher layers, because there are only 3 to 4 color channels in an image. In fact, since the first layer in that example is just gabor filters, you could manually code a program to create gabor filter units for that layer, and get massive compression. Don't know if that's true for all NN's though.
- liuliu 12y ago
  CNN layer's parameters are not interesting for compression, these are very small comparing to the massive full connect layers. In retrospective, the full connect layer's regularity probably just reflects that some neurons are more dead than others. You always probably want to have a dictionary-based compression method (LZ-like) for the last step to see how much more you can squeeze. Before that, quantization, residual quantization, some clever transformations probably can carry much longer way in terms of compressing the full connect layer parameters. I haven't explored literally any of these fancy methods yet. Just finished running xzip / gzip on best setting on my quantized full connect layer output to see if pngcrush's gain is my illusion. For the last two full connect layers, both pngcrush and xzip and gzip produce the about same size files, which suggests very limited compression ratio. On the first full connect layer, pngcrush (8.2M) is marginally better than xzip (8.6M). It is a reasonable choice as last step if you don't want to have library reference to xzip ;)
  - Houshalter 12y ago
    You could potentially compress the NN with another NN. This has been done before with HyperNEAT with success.
  - Houshalter 12y ago
    Also you can still compare neurons to see if any of them are very similar. Another thing that might work would be to prune away many of the connections. Set all of the connections that are very small to zero, or encourage it to set weights to zero with something like weight decay. As a bonus this might even improve generalization.

by liuliu 12y ago

Houshalter 12y ago

It's literally indistinguishable from random static: http://silverspaceship.com/static/shot_2.png http://silverspaceship.com/static/shot_2.png My guess is it may be making some small gains by learning "pixels are 20% more likely to be black than white" or something like that. Dictionary wouldn't work either, unless you think the NN is more likely than random to produce sequences of bits that are exactly the same. There may be regularities in deep NNs. For example, many learned features are basically exactly the same but rotated or translated in a different way. This is because NN's aren't inherently invariant to rotations or translations, so they have to learn multiple features for every possible variation. Another possible regularity is that the weights tend to cluster on specific sections of the input image and ignore the rest. But this entirely depends how the weights are projected into an image. The image provided looks like random static. The standard method is to just create a separate image for every feature, and for every pixel becomes the weights of the NN from that input pixel. Like this: http://fastml.com/images/deep-learning-made-easy/layer2visualization.png http://fastml.com/images/deep-learning-made-easy/layer2visua... Standard image compression software should be able to work with that. But this won't work in higher layers, because there are only 3 to 4 color channels in an image. In fact, since the first layer in that example is just gabor filters, you could manually code a program to create gabor filter units for that layer, and get massive compression. Don't know if that's true for all NN's though.

liuliu 12y ago
CNN layer's parameters are not interesting for compression, these are very small comparing to the massive full connect layers. In retrospective, the full connect layer's regularity probably just reflects that some neurons are more dead than others. You always probably want to have a dictionary-based compression method (LZ-like) for the last step to see how much more you can squeeze. Before that, quantization, residual quantization, some clever transformations probably can carry much longer way in terms of compressing the full connect layer parameters. I haven't explored literally any of these fancy methods yet. Just finished running xzip / gzip on best setting on my quantized full connect layer output to see if pngcrush's gain is my illusion. For the last two full connect layers, both pngcrush and xzip and gzip produce the about same size files, which suggests very limited compression ratio. On the first full connect layer, pngcrush (8.2M) is marginally better than xzip (8.6M). It is a reasonable choice as last step if you don't want to have library reference to xzip ;)
- Houshalter 12y ago
  You could potentially compress the NN with another NN. This has been done before with HyperNEAT with success.
- Houshalter 12y ago
  Also you can still compare neurons to see if any of them are very similar. Another thing that might work would be to prune away many of the connections. Set all of the connections that are very small to zero, or encourage it to set weights to zero with something like weight decay. As a bonus this might even improve generalization.