2 ms·
Llama.cpp already moves 1.5bit to the cores, they are converted in SRAM which has much higher bandwidth. FP4 only helps batched inferencing.
by xiphias2 2y ago
Llama.cpp already moves 1.5bit to the cores, they are converted in SRAM which has much higher bandwidth.
FP4 only helps batched inferencing.