3 ms·
Why run the LLM on the CPU? The 3855 chip runs opencl just fine. And there is even some viable work of it running on the NPU: https://old.reddit.com/r/Rockchi
by Havoc 2y ago
Why run the LLM on the CPU? The 3855 chip runs opencl just fine.
And there is even some viable work of it running on the NPU:
https://old.reddit.com/r/RockchipNPU/ https://old.reddit.com/r/RockchipNPU/
...that leaves the CPU free for other tasks
- rcarmo 2y agoBecause the code for that is nowhere near stable/usable unless you have the exact planetary alignment and/or blobs in your distro, and you invariably have to convert LLM weights. The number of LLMs supported is very, very small right now.
- Havoc 2y agoNPU yes. OpenCL support tends to be fine on most devices these days and has loads of weights available