3 ms·
Curious how this compares -- overall -- to the RK3588 devices that I have a few of. People have made the NPU on that thing do LLMs, and sounds like around the
by cmrdporcupine 4mo ago
Curious how this compares -- overall -- to the RK3588 devices that I have a few of.
People have made the NPU on that thing do LLMs, and sounds like around the same level (max 3Bish params, 5-6 tok/s last time I tried).
In terms of raw CPU performance, sounds slower?
But maybe has more cores?
Ouch the memory bandwidth sounds really bad.
- brucehoult 4mo agoI don't know what kind of code sysbench is using, but I get far better with a very simple `memcpy()` loop: See https://news.ycombinator.com/item?id=48523343 https://news.ycombinator.com/item?id=48523343
- LeFantome 4mo agoThe K3 is a bit slower single core, about the same multi-core, and quite a bit faster at AI. If you use the AI cores for general compute (eg. when compiling) then it is quite a bit faster than the RK3588. At times, the above may vary as specific software may be less optimized on RISC-V.