3 ms·
Well, FP4 (mostly INT4 these days) means you are moving fewer bits per weight around, so the memory bandwidth that you have performs better.
by qeternity 2y ago
Well, FP4 (mostly INT4 these days) means you are moving fewer bits per weight around, so the memory bandwidth that you have performs better.
- kccqzy 2y agoThat doesn't make sense to me. Memory bandwidth refers to the throughput of moving data from memory. Even if FP4 calculations aren't natively supported, you still move 4 bits of data per number from memory and then cast it to a FP8 or FP16 or other higher-precision number, right?
- xiphias2 2y agoLlama.cpp already moves 1.5bit to the cores, they are converted in SRAM which has much higher bandwidth. FP4 only helps batched inferencing.