6 ms·
Is there some kind of list that shows comparable performance over all of these changes? There's that ARM 4_0_X_X quant, NUMA patch, now this SDRAM tweak. Plus i
by moffkalast 2y ago
Is there some kind of list that shows comparable performance over all of these changes? There's that ARM 4_0_X_X quant, NUMA patch, now this SDRAM tweak. Plus if there's any delta with your llamafile speedups vs. openblas. I feel like we should be approaching over 3 tok/s tg for 7/8B models at 4 bits by now?
Last I tested, the ARM quants were actually slower on a Pi 5 than Q4KS, but I later learned I messed up some compile flags and haven't had the chance to retest lol.