I'm not just linking it as a proof, I now work at Intel and collaborate with Microsoft on bringing new platform technologies to life. I'm currently involved in

I'm not just linking it as a proof, I now work at Intel and collaborate with Microsoft on bringing new platform technologies to life. I'm currently involved in VNNI and hybrid core scheduling KPIs. OpenVINO allows developers to target execution engines on client based on their needs and CPU is a well supported engine for lightweight inferencing that is latency sensitive. Matmuls (FMAC especially) did evolve as a SIMD workload on CPUs but it's still forcing vector execution on cores designed around scalar workloads - you just have to be mindful of tradeoffs between executing a bunch of AVX/VNNI instructions for inferencing vs setting up a GPU/VPU/IPU context and performing that inference asynchronously. AFAIK, ARM is also interested in use of NEON/SIMD co-processors as low latency inference engines. Microsoft and Intel have a lot of interesting stuff in the works for AI compute (as I'm sure so do Apple, Nvidia and to a lesser extend AMD).