I think it's not that LLMs have redundant parameters in general - it's a specific problem with OPT-66B, not anything else. An 2022 paper "Scaling Language Mode

I think it's not that LLMs have redundant parameters in general - it's a specific problem with OPT-66B, not anything else. An 2022 paper "Scaling Language Models: Methods, Analysis & Insights from Training Gopher" (http://arxiv.org/abs/2112.11446 http://arxiv.org/abs/2112.11446) has captured it well on page 103, Appendix G: > The general finding is that whilst compressing models for a particular application has seen success, it is difficult to compress them for the objective of language modelling over a diverse corpus. The appendix G explores various techniques like pruning and distillation but found that neither method was an efficient way to obtain better loss at lower number of parameters. So why does pruning work for OPT-66B in particular? I'm not sure but there is evidence that OPT-66B is an outlier: one evidence is in the GPTQ paper ("GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers", https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323) that mentions in its footnote on its 7th page: > [2] Upon closer inspection of the OPT-66B model, it appears that this is correlated with the fact that this trained model has a significant fraction of dead units in the early layers, which may make it harder to compress. Since this article specifically target OPT-66B and not anything else, I would remain skeptical until they generalize their findings to other language models, such as Llama2 or Mistral.