4 ms·
If I wanted to run this locally at somewhat decent speeds, is an RK3588S board (like OrangePi 5) the cheapest option?
by lowbatt 1y ago
If I wanted to run this locally at somewhat decent speeds, is an RK3588S board (like OrangePi 5) the cheapest option?
- ac29 1y agoRK3588 uses a 7 year old CPU design and OrangePi 5 looks expensive (well over $100). A used sub-$100 x86 box is going to be much better
- lowbatt 1y agoyou're right. For my purposes, I was thinking of something I could use if I wanted to manufacture a new (smallish) product
- jm4 1y agoIt depends on your idea of decent speeds and what you would use it for. I just tried it on a laptop with an AMD HX 370 running on battery in power save mode and it's not especially impressive, although it runs much better in balanced or performance mode. I gave it the prompt "write a fizzbuzz program in rust" and it took almost a minute and a half. I expect it to be pretty terrible on an SBC. Your best bet is to try it out on the oldest hardware you have and figure out if you can tolerate worse performance.
- lowbatt 1y agogood idea, will test that out
- babl-yc 1y agoI'm going to attempt to get it running on the BeagleY-AI https://www.beagleboard.org/boards/beagley-ai https://www.beagleboard.org/boards/beagley-ai Similar form factor to raspberry pi but with 4 TOPS of performance and enough RAM.
- zipping1549 1y agoTried with S25+ (SD 8 elite). 0.82tok/s(4B L model). It's barely useful speed but it's pretty impressive either.
- ThatPlayer 1y agoThe RK3588 is a bit interesting because of its NPU. You can find models that have been converted to take advantage of that on HuggingFace: https://huggingface.co/models?search=rk3588 https://huggingface.co/models?search=rk3588 . No clue how performance compares. Not sure it's worth dealing with the lesser software support compared to getting an AMD mini PC and using Vulkan on llama.cpp for standard GGUF models.