4 ms·
I don't believe that quantization comes for free. Someone made the observation that llama3 models quantize "worse" than llama2 models do -- they suffer in quali
by mlsu 2y ago
I don't believe that quantization comes for free. Someone made the observation that llama3 models quantize "worse" than llama2 models do -- they suffer in quality from quantization far more.
My intuition is that a model which is undertrained suffers less from quantization, because the training process has not utilized each weight to its full potential. One of the key findings with llama, and why it punches above its weight for its size, is that they trained it for longer on a much larger dataset then was "optimal" according to the literature up to that point.
Putting two and two together, it seems that:
small model, lots of data, long training > large model + quantization
That basically, quantization is a lossy shortcut to the tail of training long. Amount and quality of data is, as always, the most important part about all of this.
- kir-gadjello 2y agoWhile llama3-8b might be slightly more brittle under quantization, llama3-70b really surprised myself and others[1] in how well it performs even in the 2..3 bits per parameter regime. It requires one of the most advanced quantization methods (IQ2_XS specifically) to work, but the reward is a SoTA LLM that fits on one 4090 GPU with 8K context (KV-cache uncompressed btw) and allows for advanced usecases such as powering the agent engine I'm working on: https://github.com/kir-gadjello/picoagent-rnd https://github.com/kir-gadjello/picoagent-rnd For me it completely replaced strong models such as Mixtral-8x7B and DeepSeek-Coder-Instruct-33B. 1. https://www.reddit.com/r/LocalLLaMA/comments/1cst400/result_llama_3_mmlu_score_vs_quantization_for/ https://www.reddit.com/r/LocalLLaMA/comments/1cst400/result_...
- d13 2y agoHow does it compare against unequalised Llama 3 8B at 16fp? I’ve been using that locally and it’s almost replaced GPT4 for me. Runs in about 14GB of VRAM.
- LordDragonfang 2y agoWhat is your use case where you find it comparable to gpt4?
- stavros 2y agoFor creative tasks, for example, Llama 3 is much better. GPT-4 is very sterile, Llama is much more whimsical, and has a lot more character.
- CaptainOfCoit 2y ago> For creative tasks What, specifically, are you asking of these LLMs? "creative tasks" can be anything from programming to cooking recipes, so a tiny bit more specificality would be appreciated :)
- stavros 2y agoSorry, I meant making up stuff, e.g. creative writing.
- CaptainOfCoit 2y agoSo like making presentations? Composing poems? Writing novellas/novels? Technical articles?
- LordDragonfang 2y agoI've used pretty much every major LLM out there for a specific type of creative writing, and none of them are as good at it as GPT4 with the exception of maybe Claude (Opus is actually probably even better regarding the sterility). Llama 3, even 70b, is definitely not better by any measure of actual quality - it's more random, at best.
- iwontberude 2y agollama3 is nowhere near gpt4, though it is cool
- renewiltord 2y agoHoly wow. Thank you for this. Very cool. I’ve been using 8b for things it might be worth using 70b for.
- endofreach 2y ago> surprised myself and others[1] in how well it performs even in the 2..3 bits per parameter regime I am too dumb for all of this ML stuff. Can you explain what exactly that means & why it's surprising?
- m1el 2y agoArtificial neural networks work the following way: you have a bunch of “neurons” which have inputs and an output. Neuron’s inputs have weights associated with them, the larger the weight, the more influence the input has on the neuron. These weights need to be represented in our computers somehow, usually people use IEEE754 floating point numbers. But these numbers take a lot of space (32 or 16 bits). So one approach people have invented is to use more compact representation of these weights (10, 8, down to 2 bits). This process is called quantisation. Having a smaller representation makes running the model faster because models are currently limited by memory bandwidth (how long it takes to read weights from memory), going from 32 bits to 2 bits potentially leads to 16x speed up. The surprising part is that the models still produce decent results, even when a lot of information from the weights was “thrown away”.
- endofreach 2y agoOh, nice. Thank you for this explanation. Now i think i get quantisation. Very well explained for someone like me. Thank you a lot!
- acchow 2y agoQuantization isn’t a shortcut. It’s a test. More training leads to better compression. Trimming precision off the integers reveals that there was more compression that could be had.
- raverbashing 2y agoQuantization might need a 'relaxing' or 'normalizing' step before to improve results, it would be an interesting research area Quantize each block or vector as a unit, bound to some metric instead of simply quantizing the whole thing all at once. It's an interesting topic https://old.reddit.com/r/LocalLLaMA/comments/1ba55rj/overview_of_gguf_quantization_methods/ https://old.reddit.com/r/LocalLLaMA/comments/1ba55rj/overvie...
- segmondy 2y agodepends on how far. Q8's are pretty much on par with f16/f32 weights. Q5/Q6's if they drop off barely drop below 2%. folks are running Q2, Q3, etc due to being GPU poor, of course it's going to be terrible. When you want to run a 70B model that's originally 140gb and all you have is 16gb of vram.
- Xcelerate 2y agoHas anyone attempted to compress network size further by extracting the symmetry invariants of the network—e.g., those that correspond to the permutation invariance of node shufflings that leave the DAG unchanged? I did a rough calculation, and as the precision of the scalar weights decreases, the information content of the specific network permutation becomes a much higher percentage of its overall size. Depending on the particular neural network architecture, there may be other symmetries beside the symmetric group that also represent compressible redundancies.
- renonce 2y agoFor a 4096x4096 matrix its symmetry group has size 4096!. The Stirling’s approximation gives ln(4096!)=4096ln(4096)-4096 which is about 10~11 bits per 4096 numbers. This is less than 0.003 bit per parameter saved.
- deleted 2y ago[deleted]