3 ms·
Apple should mostly care about power-efficient inference I think, right? Not training. Spinning up a GPU seems like something to avoid. I mean, I wonder how th
by bee_rider 2y ago
Apple should mostly care about power-efficient inference I think, right? Not training. Spinning up a GPU seems like something to avoid.
I mean, I wonder how this thing compares to a gemm using all the cores in a cpu cluster. They might be ok with not even meeting that performance, if the accelerator can not hog all the cores and power.
At least that’s what my uninformed gut says. The workload for these things is like: little AI enhancements inside conventional apps, I think.
- lxgr 2y ago> Spinning up a GPU seems like something to avoid. You can do inference on GPUs as well, and for anything other than very small/lightweight models, such as noise cancellation or maybe speech recognition, it's probably worth the initial overhead. I believe CoreML already splits workloads between CPU, NPU, and GPU as appropriate.
- jhugo 2y agoIt’s likely not worth the additional energy usage though, at least when running on battery.
- bee_rider 2y agoYeah, this is what I was getting at. In some sense, the list of “capabilities which don’t require spinning up the GPU” is expanded. Whether something could be done by spinning up the GPU is beside the point.