3 ms·
I wonder when we begin to see the dividends of all the NPU PCs come into play. AMD have been doing some good work with their NPU/iGPU hybrid inference kernels.
by lemonish97 7mo ago
I wonder when we begin to see the dividends of all the NPU PCs come into play. AMD have been doing some good work with their NPU/iGPU hybrid inference kernels. If these larger models could be scaled down to run on NPUs, you'd see much better power advantages, compared to running them on the CPU.
- Havoc 7mo agoYou can already run some models on the NPUs in the Rockchip RK3588 SBCs which are pretty abundant. A claude 4.6 they are most certainly not, but if you get through the janky AF software ecosystem they can run small LLMs reasonably well with basically zero CPU/GPU usage
- throwa356262 7mo agoAre the NPUs really that powerful? I was under the impression that they were primarily designed for low power use.
- lemonish97 7mo agoThey seem to be getting better or more powerful. The newer Intel Panther lakes and AMD Ryzen are over 50 TOPS now, IIRC
- cheema33 7mo ago> I wonder when we begin to see the dividends of all the NPU PCs come into play. A few months ago I used Whisper from OpenAI, an automatic speech recognition system released in 2002, on my modern 20-core Intel CPU to convert audio from a video file to text. It worked fine. Took a while and the machine got hot and the fans kicked in. I then found the Intel's optimized version of whisper that used NPU. It required a lot more steps to get working, but in the end it did work and was about 6x faster. And the machine remained cool and silent in the process. Since then I have become a fan of the NPUs. They are not NVIDIA GeForce RTX 5090, but they are significantly better than a modern CPU.