7 ms·
Does FP4 support really matter if local inferencing is RAM bandwidth limited anyways? I think we should put more effort into compressing the models well to be
by xiphias2 2y ago
Does FP4 support really matter if local inferencing is RAM bandwidth limited anyways?
I think we should put more effort into compressing the models well to be able to use local GPUs better.
- qeternity 2y agoWell, FP4 (mostly INT4 these days) means you are moving fewer bits per weight around, so the memory bandwidth that you have performs better.
- kccqzy 2y agoThat doesn't make sense to me. Memory bandwidth refers to the throughput of moving data from memory. Even if FP4 calculations aren't natively supported, you still move 4 bits of data per number from memory and then cast it to a FP8 or FP16 or other higher-precision number, right?
- xiphias2 2y agoLlama.cpp already moves 1.5bit to the cores, they are converted in SRAM which has much higher bandwidth. FP4 only helps batched inferencing.