3 ms·
That is my understanding as well: low power and low latency. You can see this in action when evaluating a CoreML model on a macOS machine. The ANE takes half a
by conradev 2y ago
That is my understanding as well: low power and low latency.
You can see this in action when evaluating a CoreML model on a macOS machine. The ANE takes half as long as the GPU which takes half as long as the CPU (actual factors being model dependent)
- nickpsecurity 2y agoTo take half as long, doesn’t it have to perform twice as fast? Or am I misreading your comment?
- eightysixfour 2y agoNo, you can have latency that is independent of compute performance. The CPU/GPU may have other tasks and the work has to wait for the existing threads to finish, or for them to clock up, or have slower memory paths, etc. If you and I have the same calculator but I'm working on a set of problems and you're not, and we're both asked to do some math, it may take me longer to return it, even though the instantaneous performance of the math is the same.
- refulgentis 2y agoIn isolation, makes sense. Wouldn't it be odd for OP to present examples that are the opposite of their claim, just to get us thinking about "well the CPU is busy?" Curious for their input.
- conradev 2y agoThe GPU is stateful and requires loading shaders and initializing pipelines before doing any work. That is where its latency comes from. It is also extremely power hungry. The CPU is zero latency to get started, but takes longer because it isn't specialized at any one task and isn't massively parallel, so that is why the CPU takes even longer. The NPU often has a simpler bytecode to do more complex things like matrix multiplication implemented in hardware, rather than having to instantiate a generic compute kernel on the GPU.