8 ms·
There are now at least three ways to accelerate machine learning models on consumer hardware: - GPU compute units (used for LLMs) - GPU "neural accelerator
by cubefox 1y ago
There are now at least three ways to accelerate machine learning models on consumer hardware:
- GPU compute units (used for LLMs)
- GPU "neural accelerators"/"tensor cores" etc (used for video game anti-aliasing and increasing resolution or frame rate)
- NPUs (not sure what they are actually used for)
And of course models can also be run, without acceleration, on the CPU.
- colejohnson66 1y agoAn "NPU" is a matrix multiplier accelerator. It removes some general-purpose stuff that GPUs provide in favor of more "AI"-useful units, like support for values a byte or smaller (i.e., FP4, INT4, etc.).
- cubefox 1y agoAll three of them accelerate matrix multiplications actually.
- almostgotcaught 1y agoany thing that computes matmul faster than by hand technically accelerates matmul - so what's your point?
- MobiusHorizons 1y agoI think NPUs are often aimed at efficient matmul performance. Not all implementations are significantly faster than vector units in the CPU, but they use much lower power. Gpu acceleration is typically much faster than the CPU, but also higher power.
- atq2119 1y agoAt least for desktop gaming, the tensor cores are in the GPU compute units (SM), same as for the big data center GPUs. It seems ARM believe it makes sense to go a different route for mobile gaming.
- catgary 1y agoFrom what I can tell, NPUs are mostly being used by Microsoft to encourage vendor lock-in to the MicrosoftML/ONNX platform (similar to their DirectX playbook).
- jonas21 1y agoThey're used a lot on mobile. Apple uses their "neural engine" NPU to power their on-device ML stuff and Samsung does something similar in their Exynos processors. Apple also exposes the NPU to developers via CoreML.
- pjmlp 1y agoExtension spaghetti is hardly any better, just because it says the same API name on the tin. Google and Apple have been doing NPUs for a while now.
- bigyabai 1y agoYeah, and the NPUs have displaced approximately 0% of the hardware demand for Nvidia products. Real snipe chase they've commit to. Extension spaghetti is fine, I'd much rather end up with AI acceleration being handled like Vulkan than suffering a fate like Metal or DirectX.
- pjmlp 1y agoFailure to understand why developers chose CUDA, is exactly why NVidia keeps selling. Same applies to proprietary 3D APIs. There is a reason why only FOSS devs make such big fuss out of APIs, while professional game studios keep talk about how to take each hardware to its limits at GDC, since 8 bit heterogeneous home game systems.