8 ms·
Self-Compressing Neural Networks
- bilsbie 2y agodynamic quantization-aware training that puts size (in bytes) of the model in the loss
- diimdeep 2y agoI know where you saw that https://x.com/realGeorgeHotz/status/1819963680739512550 https://x.com/realGeorgeHotz/status/1819963680739512550 > This is one of the coolest papers I've seen in a while. "Self-Compressing Neural Networks" is dynamic quantization-aware training that puts size (in bytes) of the model in the loss! > My implementation (in @__tinygrad__): https://github.com/geohot/ai-notebooks/blob/master/mnist_self_compression.ipynb https://github.com/geohot/ai-notebooks/blob/master/mnist_sel...
- bilsbie 2y agoTrue. It seemed like a good, concise summary. Maybe I should have credited him.
- mlajtos 2y agoThat is pretty cool. I found a follow up work that applies this technique to LLM: https://konczer.github.io/doc/Poster_EEML23_JozsefKonczer.pdf https://konczer.github.io/doc/Poster_EEML23_JozsefKonczer.pd...
- andrewflnr 2y agoThis kind of thing, much more than LLMs, makes me worry about AGI takeoff.
- tazu 2y agoWhy? Do you think lossless compression is intelligence?
- luckystarr 2y agoParents thinking was probably: If you can achieve similar results with a fraction of memory/compute usage then capability at the same hardware level will increase even more.
- andrewflnr 2y agoIt's specifically the fact that the network is directing its own optimization. Which yes, could then potentially be used to get more capability from the hardware, but that's true of manually optimized networks as well. Needing less human help is the... interesting part.
- HeatrayEnjoyer 2y ago"Hardware overhang" is the term of art. My meek opinion is this is obvious. Human-level intelligence requires at most 20 watts and substrate no more complicated than can be constructed from simple organic molecules in a dirty environment. What is possible with 20 kilowatts and wafer fabricators?
- andrewflnr 2y agoNo, but since you mentioned it: https://en.wikipedia.org/wiki/Hutter_Prize https://en.wikipedia.org/wiki/Hutter_Prize Anyway, OP is about lossy compression. I can't fully follow it but they talk about techniques for mitigating loss later in the paper.
- idiotsecant 2y agoCompressing understanding (not just information) in a way that uses semantic links in information is a big part of intelligence, I'd say.
- visarga 2y agoWe're doing a double search - searching for experience outside, collecting data - and searching for understanding inside, by compressing the data. Search and learn, they define both AI and us.
- throwup238 2y agoI think this might be the first step to making neural networks that actually mimic biological brains. IMO the biggest piece missing from NN architectures is a mechanism like neuroplasticity that modifies the topology of neurons. Brains reorganize themselves around the things they learn. This paper is a long way from implementing synaptic pruning/strengthening/weakening, neurogenesis, or synaptogenesis but it’s the first one I’ve seen where the network is self optimizing.
- wigster 2y agoi know nothing of which i speak.... but the theme reminded me of the insect brains, with very relatively few Neurons that manage pretty extraordinary feats. i guess random evolution pruning happens and if there is no detrimental effect, cheerio.
- nyrikki 2y agoUnfortunately dendritic compartmentalization, spike timing etc are still not present. All efforts at models of SNNs that I know of have hit problems like riddled basins so far, that is what to look for to move past the limits of perceptron based networks IMHO. As PAC learning with autograd and perceptrons is just compression, or set shattering, this paper is more of an optimization method that reduces ANN expressiveness through additional compression. Being able to control loss of precision is exciting though. It may help in some cases, especially for practical use cases, but their unaddressed mention of potential problems with noisy loss functions needs to be addressed. Human biological neurons can do XOR in the dendrites without hitting the soma at all is another example. If you haven't heard about dendritic compartmentalization and plasticity, here is a paper. https://www.cell.com/neuron/fulltext/S0896-6273(11)00993-7 https://www.cell.com/neuron/fulltext/S0896-6273(11)00993-7 > In conclusion our results support the view that experience can drive clustered synaptic enhancement onto neuronal dendritic subcompartments, providing fundamental architecture to circuit development and function
- derefr 2y ago> reduces ANN expressiveness But does it? It’s been my hypothesis for a while that every grad-trained NN is hauling around a lot of “nascent” nodes — nodes that were on their way to being useful, but haven’t received enough input yet to actually have their outputs be distinguishable from noise / ever influence the output. Sort of the neuroplastic equivalent of an evolutionary pre-adaptation. If such nodes exist in NNs, they would be important to decreasing training time to learning new concepts given further training; but if there will be no more training, then they could be pruned for literally no change in expressivity (i.e. the optimality of the NN as an autoencoder of the existing training data.)
- Version467 2y agoThis is super cool. It's surprising to me that it took so long for someone to try this. It seems like such an obvious idea (in hindsight). But I guess that's easy to say now that someone came up with it. If this turns out to work well even for much larger models, then we might see loss functions that incorporate ever more specific performance metrics, conceivably even actual execution times on specific hardware.
- xpe 2y agoThere was related work that happened before, as mentioned in the paper.
- Version467 2y agoWhoops, missed that. Thanks.
- szcs 2y agoAuthor here. I actually came up with the idea a long time ago, I first experimented with variants of this in Caffe (before Tensorflow was a thing).
- spacemanspiff01 2y agoSo this was published a year and a half ago? Is there a reason it did not catch on?
- svantana 2y agoIt's not really that innovative. As the paper notes, there are several similar previous works. Also, it sounds like they have done a bunch of tweaking to reduce the "irreversible forgetting" specifically for this particular dataset and network, which is not very scientific. Further testing is required to see if this method really has legs.
- deleted 2y ago[deleted]
- w-m 2y agoUsing as little computational resources (memory and/or FLOPS) as possible as an additional optimization criterion when training NNs is an interesting avenue. I think the current state of pre-trained model families is weird. Take Llama 3.1 or Segment Anything 2: you get tiny/small/medium/larger/huge models, where for each tier the model size was predefined, and they are trained somewhat (completely?) independently. This feels iffy, patchy, and like we haven't really arrived yet. I'd want a model that scales up and down depending on the task given at inference, and a model that doesn't have a fixed size when starting the training. Shouldn't it specialize over training progress, when seeing more tokens, and grow larger where needed? Without some human fixing a size beforehand? Self-organization is a fascinating topic to me. This last year I've been working on Self-Organizing Gaussian Splats [0]. With a lot of squinting, this lives in a similar space as the Self-Compressing Neural Networks from the link above. The idea of the Gaussians was to build on Self-Organizing Maps (lovely 90s concept, look for some GIFs if you don't know it), and use that to represent 3D scenes in a memory-efficient way. By mapping attributes into a locally smooth 2D grid. It's quite a simple algorithm, but works really well, and better than many quite complicated coding schemes. So this has me excited that we'll (re-)discover great methods in this space in the near future. [0]: https://fraunhoferhhi.github.io/Self-Organizing-Gaussians/ https://fraunhoferhhi.github.io/Self-Organizing-Gaussians/
- idontknowmuch 2y agoAfaik, they aren't really trained independently -- for most models, e.g. DINO, etc., the smaller sizes are actually distilled from larger models. It's much easier to generate performant models at smaller size via distillation. And I'd be curious of the utility of model that scales up and down at inference - if this was the case you'd still need to have storage that is the same as the maximum model size. This would essentially be useless for embedded applications, etc., unless you have heavy quantization - but quantization in a small parameter space would probably make the smaller modes useless. I could see the benefit here in terms of optimizing latency for different applications but maybe you have other ideas. Given all that, I think training for smaller number of parameters, as noted in OP, would kind of beat out some model that scales at inference time - especially when most people know what kind of application they are aiming to build and the required level of performance.
- octocop 2y agoWhat advantages does this have over applying neural network compression methods after training?
- Jensson 2y agoIf you do it during training you automatically check that the compressed model still do what you want after compression, and can even correct some small issues by continuing the training on the smaller model. Edit: In general the more a compression function understands of what your goals are the better, so it is naturally advantageous to make the compression function look like training since then it is fully aware of what to optimize for.
- szcs 2y agoHere is a quick graph I just generated from data I had. Horizontal axis is average bit depth, vertical is accuracy. PTQ is Post Training Quantisation, QAT is Quantisation Aware Training. It's a simple ResNet trained on CIFAR-10, I don't have the resources to train anything bigger. https://i.imgur.com/yiVLljh.png https://i.imgur.com/yiVLljh.png
- szcs 2y agoAuthor here, I just noticed this. If you have any questions I can try answering them.
- smusamashah 2y agoDo you have plans to apply this on a small open LLM as a POC to show that there is no loss of performance?
- szcs 2y agoYes, LLMs are planned too.
- fooblaster 2y agoI don't think I quite understand how to apply your technique to gpus, which have discrete formats like fp4, fp8, fp16, int8, etc. Can you shed some light on that?
- Gettingolderev 2y agoAnyone knows if re-ordering also helps here? If i can sort the lines of the matrix, which is probably defined by how the token embedding is setup, i could potentially zero out weights which do not contribute at all and have areas of zeros i could mark and skip?
- dpkingma 2y agoThere seems to be some relevant prior work that is not referenced by this paper, such as our work on training sparse neural networks: https://arxiv.org/abs/1712.01312 https://arxiv.org/abs/1712.01312 Abstract: "We propose a practical method for L0 norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. [...]"
- alecco 2y ago(Jan 2023)
- adipandas 2y agoBeautiful idea. My take on it: I find it difficult to generalize the notion of layer removal when the bit depth of that layer goes to zero. It's wouldn't be straight forward although the authors provide equation 5. It feels like lot of information is missing in this work to even reproduce it. And authors do only 1 case study. I believe some implementation is required to understand the authors completely. Example, optimizer modification for layer when it is removed in training.
- smusamashah 2y agoBlog post on the subject by author https://blog.imaginationtech.com/self-compressing-neural-networks https://blog.imaginationtech.com/self-compressing-neural-net...