10 ms·
Orange Pi 5 has an NPU. I wonder if it'd be any faster than using GPU.
by jan_Sate 3y ago
Orange Pi 5 has an NPU. I wonder if it'd be any faster than using GPU.
- deleted 3y ago[deleted]
- adrian_b 3y agoIt might have been more work to convert the model for the RK3588 NPU, even if Rockchip provides an SDK and an automated conversion tool that should help (the SDK includes a simulator for the NPU, so the converted model can be tried on a PC before being deployed on a board like Orange Pi): https://wiki.t-firefly.com/en/ROC-RK3588S-PC/usage_npu.html https://wiki.t-firefly.com/en/ROC-RK3588S-PC/usage_npu.html
- brucethemoose2 3y agoIts not supported by TVM yet, but there is support for Qualcomm Hexagon. You can kinda see some of the supported backends gated behind flags in the cmake file: https://github.com/mlc-ai/relax/blob/mlc/CMakeLists.txt https://github.com/mlc-ai/relax/blob/mlc/CMakeLists.txt
- duchenne 3y agoHow many tokens per second do you think we can get out of this 6TFlops NPU?
- brucethemoose2 3y agoFor prompt ingestion... I dunno. Unbatched token generation is basically RAM bandwidth limited, as the entire model has to be cycled through for each token. I bet theoretical performance is similar to the GPU, albeit with much lower power consumption.