43 ms·
SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
- Reubend 4y agoWow, these results are amazing! This might be extremely helpful in reducing the memory consumption of large models in the future.
- avereveard 4y agowow this could make the fabled 65 billion parameter llama sparsed and pruned runnable on a 3060
- bick_nyers 4y agoHow would this be any different from running one of the lower parameter models?
- sva_ 4y agoIt says in the abstract > at minimal loss of accuracy Suggesting that there is a lot of redundancy in the weights.
- theLiminator 4y agoI wonder how far we can take this. Is 1B parameters theoretically "expressive" enough for GPT-4 like performance? I wonder how far off "theoretically optimal" we are in terms of performance/parameters ratio.
- dontwearitout 4y agoGood open questions. I suspect we'll see models distilled and compressed down to retain most of their "common sense", "core knowledge", and reasoning ability, while stripping out the gigabytes of random trivia most people will never need.
- amelius 4y agoThis makes me wonder what would happen if you took the sparse model, reset its weights, then trained it with the original training data.
- sva_ 4y agoVery interesting idea. I'd hypothesize that it won't achieve the same(ish) accuracy, and that pruning might be required (similar to how humans go through a heavy pruning phase at an early age[0]). Would be worth setting up an experiment on a smaller scale. As some other commentator stated, there's currently a lot of low hanging fruit in optimizing NN. 0. https://en.m.wikipedia.org/wiki/Synaptic_pruning https://en.m.wikipedia.org/wiki/Synaptic_pruning
- avereveard 4y agoLarger models seem to handle much better introspection, makes for better backend for sourced knowledge extraction
- babyshake 4y agoIt looks like by pruning by a factor of 0.5, you reduce the size of the model by 50%? In practice, what is the expected observed change in the output before and after pruning?
- MacsHeadroom 4y ago>what is the expected observed change in the output before and after pruning? The expected and observed change is virtually none. That's the whole point! Notably, quantizing weights from 16bit weights to 4bit weights (reducing the size by 75%) also has almost no change in output quality when using modern algorithms like GPTQ.
- levidos 4y agoWhat are the drawbacks of quantizing though?
- huslage 4y agoYou are losing information. Knowing if you need that information or not is sometimes difficult. That's the whole lottery hypothesis (mentioned elsewhere).
- MacsHeadroom 4y ago>What are the drawbacks of quantizing though? A 0.01% loss in quality for a 4x speed up and 4x less VRAM/RAM requirement. 90GB models now fit and run on a $600 consumer video card with quality so similar the difference is only detectable on hours long automated tests with tens of thousands of iterations.
- gorkish 4y agoVery good result, and awesome to see such great progress happen so fast. Quantizing/pruning/deduplicating/compressing models and embeddings is still a vast orchard of low hanging fruit. I personally think there are still quite a few multiple-orders-of-magnitude scale opportunities to accelerate inference, and we are fortunate to have strong economic incentives aligned with the problem.
- Der_Einzige 4y agoSo much low hanging fruit for those willing to pick it. It's a great time to be an LLM researcher.
- bitL 4y agoYeah, writing a thesis on it right now; this + adapters give so many options to play with and Meta was nice to give me access to their research models.
- seydor 4y agoThe robustness with which these models can be quantized, and now trimmed makes one think if they could be easily implemented some form of analog (or optical) hardware.
- meepmorp 4y agoIsn't the inspiration behind these models our own analog hardware?
- seydor 4y agonot sure i would call neurons analog, they are very nonlinear and capricious beasts.
- mitthrowaway2 4y agoPerhaps even by biological cells!
- barking_biscuit 4y agoXenobots, even.
- dougmwne 4y agoOnce you prune your model, can you get even better performance by re-training it? I've heard theories that this is the function of sleep in brains.
- napo 4y agoIt sounds nice (maybe too nice?). I always wanted to see that it would be necessary to have a "sleeping" phase in AI. It always felt weird that we have to sleep, it doesn't seem to give any evolutionary advantages.
- roomey 4y agoIt must give an evolutionary advantage, or we wouldn't sleep. It may be hard to pin point exactly what advantage, but as we do it, it must have given us an advantage!
- dougmwne 4y agoEspecially considering that it is so widespread in nearly every creature with a brain. And it’s not simply a period of motionless energy conservation but has very specific neural patterns. The science is definitely zeroing in on a connection to learning.
- richardw 4y agoI have an unbaked theory, but the very short version is: - Animals that have peaks of energy use outcompete animals that have a steady-state energy use. Catch the animal, then rest and recover. For any given amount of energy, this means we can recruit more in a smaller window compared to an animal that plods along with no recuperative phase. - Many things happen when you're sleeping. Rather than having everything running 24/7, having different phases means we can specialise action and recovery. Since the time is already driven by energy demands, many parts of our body and mind leverage it for different purposes.
- LesZedCB 4y ago
- smrtinsert 4y agoTo copy a reddit meme: text-generation-webui plugin when? But seriously, this seems like an incredible upgrade.
- chime 4y agoBetter yet, text-generation-webui-docker when?
- NortySpock 4y agoIs Serge what you are looking for? I've been using it to play around with prompting a few large language models. https://github.com/nsarrazin/serge https://github.com/nsarrazin/serge
- vessenes 4y agoThis is interesting. OPT and BLOOM are significantly Chinchilla under-trained, and I can't help but wonder if this is related to their compressibility here. I would like to see the results for something Chinchilla over-trained, like Llama - my gut is that the 'free lunch' they see will get slightly more expensive. Implementation q - can torch or other inference runtimes take advantage of the memory savings delivered by a sparsification like this? Or do you need a special implementation to not malloc out all the memory implied by each tensor layer?
- MacsHeadroom 4y agoSparseGPT-for-LLaMA[0] exists. Pruning more than 30% of weights of 30B starts to show significant perplexity losses. 50% is a total disaster, while it was not for OPT or BLOOM. So your intuition seems to be good here. [0] https://github.com/AlpinDale/sparsegpt-for-LLaMA https://github.com/AlpinDale/sparsegpt-for-LLaMA
- generalizations 4y agoHas anyone figured out what the optimal pruning is for 65b? I don't really know what that matrix in your link is saying, but it didn't seem to show optimal pruning.
- nalzok 4y agoCould you explain what you mean by "Chinchilla under-trained" or "Chinchilla over-trained"? I assume it refers to some measure of trained-ness, but Googling yielded nothing relevant.
- wlib 4y agoMy memory says that there’s a “Chinchilla” paper showing how to make the best model with a given training budget. There’s a trade-off between the amount of training data and the size of the model itself. Chinchilla under-training would mean that the model is too big for the amount of training data used. Llama is Chinchilla over-trained in that there is a ton of data relative to the small size of the model. Note that this is still desirable for inference because you want the most possible training on whatever model you can actually fit in your memory.
- valine 4y agoIf the abstract is accurate, then I'm very, very excited to try this on LLaMA 65B. We are tantalizingly close to ChatGPT performance parity on consumer hardware. Hopefully this lowers the cost of doing instruct fine tuning on the larger models, and we see a Vicuna like model based on LLaMA 65B soon. This is exciting folks.
- icyfox 4y agoNeat paper. Planning on reading more in-depth over the weekend, but more fundamental than just applications to GPT their insights are: - Existing pruners were written for models that are order-of-magnitudes smaller than any in the modern GPT family. They grow in linear time with the amount of input parameters so they're unequipped to work on current architectures. The best existing pruner performs takes 4.3h for a 1.3B model - The core issue to scale is time to calculate the Hessian during prune analysis (effectively a matrix of second-order derivatives, famously computationally intense to calculate) - They follow the existing literature and use a local approach to each layer. By doing this (and doing it well), it can preserve the input/output contract for surrounding layers, which makes the whole thing paralellizable across machines - Their solution approximates reconstruction loss by approximating a quadratic loss and then running a OBS update (with a few other optimizations on ordering and iteration on the side) I'm particularly excited for these smaller models, mostly for inference efficiency gains in realtime applications. The general con of weight pruning is they still require incredibly large training clusters / investment in training resources upfront to get the original parameter weight. But if the lottery ticket hypothesis holds true, this might be the best way we have at the moment to get models with same performance and lower longterm operational costs.
- icyfox 4y agoAnother random thought: Most of these general purpose pruning approaches rely on randomly calculating the X vector for which they want to measure output loss of the layer. In theory it's possible to feed actual datasets into these models as well, which could be another way to get a sparse model that's more acutely optimized towards one task. The original model produces the X activations on each layer, and these are used as the optimization criteria for the pruned version. It might be able to provide performance similar to fine-tuning but without the weight skew that you'll necessarily see in parameter values.
- CuriouslyC 4y agoStatisticians have been using L1 regularization to estimate sparse models for a while, it seems reasonable to assume that you could fine tune the model on a data set while also pushing weak parameters to zero in a natural way in this domain as well.
- m3kw9 4y agoWho’s gonna apply this to LLama weights?
- MacsHeadroom 4y agoIt was done over two months ago: https://github.com/AlpinDale/sparsegpt-for-LLaMA https://github.com/AlpinDale/sparsegpt-for-LLaMA
- Jimmc414 4y agoEternal Sunshine of the Spotless Mind for LLMs.
- ben_w 4y agoI don't think so, from the abstract it's more like JPEG for LLMs.
- igravious 4y agoOne hopes it's more like PNG for LLMs ?
- ben_w 4y agoIt's lossy rather than lossless, and the impact can be dialled up or down depending on space requirements.
- barking_biscuit 4y agoCan we call it LLMPeg?
- NM_Ricky 4y agoFor anyone interested in SparseGPT, on May 25th, the author of the SparseGPT paper will show you how you can download an optimized and open-sourced LLM and run it on CPUs at GPU speeds using DeepSparse. Confirm your spot: https://neuralmagic.com/unlock-faster-and-more-efficient-language-models-with-sparsegpt/ https://neuralmagic.com/unlock-faster-and-more-efficient-lan...
- deleted 4y ago[deleted]
- laughy 4y agoThis suggests that the effective number of parameters is far lower than the nominal number. My head canon for neural networks as overparametrized models still holds.