4 ms·
It looks like by pruning by a factor of 0.5, you reduce the size of the model by 50%? In practice, what is the expected observed change in the output before and
by babyshake 3y ago
It looks like by pruning by a factor of 0.5, you reduce the size of the model by 50%? In practice, what is the expected observed change in the output before and after pruning?
- MacsHeadroom 3y ago>what is the expected observed change in the output before and after pruning? The expected and observed change is virtually none. That's the whole point! Notably, quantizing weights from 16bit weights to 4bit weights (reducing the size by 75%) also has almost no change in output quality when using modern algorithms like GPTQ.
- levidos 3y agoWhat are the drawbacks of quantizing though?
- huslage 3y agoYou are losing information. Knowing if you need that information or not is sometimes difficult. That's the whole lottery hypothesis (mentioned elsewhere).
- MacsHeadroom 3y ago>What are the drawbacks of quantizing though? A 0.01% loss in quality for a 4x speed up and 4x less VRAM/RAM requirement. 90GB models now fit and run on a $600 consumer video card with quality so similar the difference is only detectable on hours long automated tests with tens of thousands of iterations.