3 ms·
What are you doing with local llms on a pi5, genuine question btw.
by TechDebtDevin 2y ago
What are you doing with local llms on a pi5, genuine question btw.
- numberless 2y agoNot the author, but on a Pi 5 small LLMs such as tinyllama or qwen2.5:0.5b run at over 25 tokens per second (haven't tested the performance boost with it yet). It could be useful if you wanted a local assistant or an AI server for devices you don't want to run LLMs on
- tmikaeld 2y agoJust curious, what can a 0.5B model be used for?
- tetris11 2y agoI use mine mostly as a context-free code copilot. It's not perfect, but it knows how to write a template and for most tasks that's all I need.
- nirav72 2y agoAre there any vision models that will work on the Pi 5? Something similar to minicpm-v?
- menaerus 2y agoI don't believe you can get any useable results. Perhaps only for R&D.