4 ms·
> Quantization creates some brain damage. Love the wording.
by FrostKiwi 2y ago
> Quantization creates some brain damage.
Love the wording.
- AMICABoard 2y agoMaybe we should make "Brain Damage Factor" a official term to denote how much types of quantizations degrade output compared to unquantized.:)
- Y_Y 2y agoI think that's known as degradation, but I think the brain damage metric could be usefully applies to the poor sods who try to get the quantization to work in the first place.
- AMICABoard 2y agoLol :)
- impossiblefork 2y agoThere's actually an old paper titled Optimal Brain Damage, where they don't try to find optimal quantizations, but optimal sparse versions of a models-- i.e. where some weights are set to zero.
- AMICABoard 2y agoInteresting, I heard something like that, but now I must read about it.
- nickpsecurity 2y agoI found… Optimal Brain Damage https://www.researchgate.net/publication/221618539_Optimal_Brain_Damage https://www.researchgate.net/publication/221618539_Optimal_B... Optimal Brain Compression https://openreview.net/pdf?id=ksVGCOlOEba#:~:text=The%20resulting%20algorithm%2C%20called%20the,weights%2C%20further%20reducing%20the%20loss https://openreview.net/pdf?id=ksVGCOlOEba#:~:text=The%20resu.... TinyVolt’s implementation of it: https://github.com/TinyVolt/optimal-brain-compression https://github.com/TinyVolt/optimal-brain-compression
- AMICABoard 2y agoThanks, I'll read it up. Interesting.
- nickpsecurity 2y agoThat was really for them. You’re out there building neat stuff. Your talent might warrant looking into AdderNets and Bitnets which might get the cost down. There’s also some brain-inspired designs. I don’t think many people have implemented such things. You might discover something new experimenting with them.
- AMICABoard 2y ago:)