3 ms·
For me, the most interesting part is the statistics on all the models. These show that 8 bit quantization is basically as good as the full model and 4 bit is ve
by Loic 4y ago
For me, the most interesting part is the statistics on all the models. These show that 8 bit quantization is basically as good as the full model and 4 bit is very close. This is the first time I see such table across a large number of models in one place.
- recuter 4y agoPretty much. Llama specific: https://github.com/qwopqwop200/GPTQ-for-LLaMa https://github.com/qwopqwop200/GPTQ-for-LLaMa > According to GPTQ paper, As the size of the model increases, the difference in performance between FP16 and GPTQ decreases. https://nolanoorg.substack.com/p/int-4-llama-is-not-enough-int-3-and https://nolanoorg.substack.com/p/int-4-llama-is-not-enough-i... https://docs.google.com/document/d/1wZ0g9rHI-6s7ctNlykuK4W5T-XijQXlZivY7_s-JR1E/edit https://docs.google.com/document/d/1wZ0g9rHI-6s7ctNlykuK4W5T... Expect to get away with a factor of 4-5 reduction in memory usage for a minimal loss of quality. :)