3 ms·
Just tried the SDRAM tweak and got myself a respectable 10% speedup with LLMs. Hat tip.
by jart 2y ago
Just tried the SDRAM tweak and got myself a respectable 10% speedup with LLMs. Hat tip.
- TechDebtDevin 2y agoWhat are you doing with local llms on a pi5, genuine question btw.
- numberless 2y agoNot the author, but on a Pi 5 small LLMs such as tinyllama or qwen2.5:0.5b run at over 25 tokens per second (haven't tested the performance boost with it yet). It could be useful if you wanted a local assistant or an AI server for devices you don't want to run LLMs on
- tmikaeld 2y agoJust curious, what can a 0.5B model be used for?
- tetris11 2y agoI use mine mostly as a context-free code copilot. It's not perfect, but it knows how to write a template and for most tasks that's all I need.
- nirav72 2y agoAre there any vision models that will work on the Pi 5? Something similar to minicpm-v?
- menaerus 2y agoI don't believe you can get any useable results. Perhaps only for R&D.
- moffkalast 2y agoIs there some kind of list that shows comparable performance over all of these changes? There's that ARM 4_0_X_X quant, NUMA patch, now this SDRAM tweak. Plus if there's any delta with your llamafile speedups vs. openblas. I feel like we should be approaching over 3 tok/s tg for 7/8B models at 4 bits by now? Last I tested, the ARM quants were actually slower on a Pi 5 than Q4KS, but I later learned I messed up some compile flags and haven't had the chance to retest lol.