32 ms·

The prices of data center CPUs are also quite eye watering. And their compute throughput is one to two orders of magnitude lower than that of a modern, high end

• adrian_b 3y ago
  "One to two orders of magnitude" may be true only for FP32 or FP16 and only when comparing a single GPU to a single CPU. For FP64, the ratio in the performance per watt between the few datacenter-only GPU models that support FP64 and CPUs has been only around 3 for multiple GPU and CPU generations. For inference using the AMX instruction set I have not seen any useful published benchmark that can show which is the ratio between the performance per watt of Emerald Rapids vs. NVIDIA, but the performance per dollar is certainly much worse for NVIDIA. What limits the performance is normally either the performance per dollar or the performance per watt, it seldom matters if the performance is provided by a single huge GPU or by multiple smaller GPUs or CPUs. While the top models of the server CPUs are overpriced, there are also models with very decent prices, like the 28-core Xeon 5512U @ $1230 or AMD Ryzen 9 7950X.
  • ein0p 3y ago
    Nobody uses FP64 for AI. Nobody uses consumer SKUs in the datacenters.