5 ms·

CPUs are actually involved, especially with SIMD like AVX and VNNI instructions. It all depends on the scale of your compute. https://www.intel.com/content/www

• refulgentis 3y ago
  Linking to an Intel SDK with “deep learning” in it does prove it’s possible to do matmuls on CPUs. But this a nitpicked to death thread: it’s good to see Apple releasing research, the Metal stuff was to make it run on their GPU, and there simply aren’t any relevant uses of CPUs. Academically, yes, CPUs have SIMD so they can implement the same algorithm that drives GPU perf, but without some massive breakthrough the one that can do more matmuls wins, and nVidia wins massively, in theory and practice.
  • zeusk 3y ago
    I'm not just linking it as a proof, I now work at Intel and collaborate with Microsoft on bringing new platform technologies to life. I'm currently involved in VNNI and hybrid core scheduling KPIs. OpenVINO allows developers to target execution engines on client based on their needs and CPU is a well supported engine for lightweight inferencing that is latency sensitive. Matmuls (FMAC especially) did evolve as a SIMD workload on CPUs but it's still forcing vector execution on cores designed around scalar workloads - you just have to be mindful of tradeoffs between executing a bunch of AVX/VNNI instructions for inferencing vs setting up a GPU/VPU/IPU context and performing that inference asynchronously. AFAIK, ARM is also interested in use of NEON/SIMD co-processors as low latency inference engines. Microsoft and Intel have a lot of interesting stuff in the works for AI compute (as I'm sure so do Apple, Nvidia and to a lesser extend AMD).