4 ms·
curiously, what real world usage actually uses FP4? AFICT, most of the LLMs still use BF16, and even the quantizations down to 4bits and 2bits end up back to 16
by jxy 3y ago
curiously, what real world usage actually uses FP4? AFICT, most of the LLMs still use BF16, and even the quantizations down to 4bits and 2bits end up back to 16bit or INT8 for actual computations.
- buildbot 3y agoLlama.cpp and many others support 4 bit weights and lower
- creshal 3y agoHalf the reason why they move back up to 8/16 bit is that current hardware doesn't properly support 4 bit floats, and you get better performance from the conversion. I think once this hardware hits, most of the computation will shift to native 4 bit just for efficency's sake. ...assuming the recent 1.58b paper doesn't render the entire float quantization approach obsolete by then.
- imtringued 3y agoThe 1.58b approach is good for everyone including for quantization. It means that current quantization schemes have room for improvement.
- creshal 3y agoIt's good for everyone on the software side, but it's not so good for the hardware side, because it means that whatever you're designing now for tapeout in 6 months and release in 12 is going to be obsolete in 2 weeks.