4 ms·
There's actually an old paper titled Optimal Brain Damage, where they don't try to find optimal quantizations, but optimal sparse versions of a models-- i.e. wh
by impossiblefork 2y ago
There's actually an old paper titled Optimal Brain Damage, where they don't try to find optimal quantizations, but optimal sparse versions of a models-- i.e. where some weights are set to zero.
- AMICABoard 2y agoInteresting, I heard something like that, but now I must read about it.
- nickpsecurity 2y agoI found… Optimal Brain Damage https://www.researchgate.net/publication/221618539_Optimal_Brain_Damage https://www.researchgate.net/publication/221618539_Optimal_B... Optimal Brain Compression https://openreview.net/pdf?id=ksVGCOlOEba#:~:text=The%20resulting%20algorithm%2C%20called%20the,weights%2C%20further%20reducing%20the%20loss https://openreview.net/pdf?id=ksVGCOlOEba#:~:text=The%20resu.... TinyVolt’s implementation of it: https://github.com/TinyVolt/optimal-brain-compression https://github.com/TinyVolt/optimal-brain-compression
- AMICABoard 2y agoThanks, I'll read it up. Interesting.
- nickpsecurity 2y agoThat was really for them. You’re out there building neat stuff. Your talent might warrant looking into AdderNets and Bitnets which might get the cost down. There’s also some brain-inspired designs. I don’t think many people have implemented such things. You might discover something new experimenting with them.
- AMICABoard 2y ago:)