2 ms·
There's also research showing that the perplexity reduction is less at higher parameter counts. E.g. a 65b parameter model barely has any impact at all when red
by Taek 3y ago
There's also research showing that the perplexity reduction is less at higher parameter counts. E.g. a 65b parameter model barely has any impact at all when reducing from 16bit to 4bit
- Ambix 3y agoNot actually, you might see here how bigger models have much worse perplexity with 4bit due to the weight outliers: https://github.com/saharNooby/rwkv.cpp/issues/12 https://github.com/saharNooby/rwkv.cpp/issues/12 For LLaMA models - yeah, different story.