3 ms·
That would be true, but the prices for datacenter GPUs are so inflated (both for NVDIA and for AMD) that it does not matter that CPUs are not as efficient from
by adrian_b 3y ago
That would be true, but the prices for datacenter GPUs are so inflated (both for NVDIA and for AMD) that it does not matter that CPUs are not as efficient from the technical point of view (i.e. in performance per watt and performance per die area), because the performance per dollar of CPUs is currently much better, unlike 10 years ago, when the GPUs were much cheaper, so they also had better performance per dollar.
- ein0p 3y agoThe prices of data center CPUs are also quite eye watering. And their compute throughput is one to two orders of magnitude lower than that of a modern, high end GPU.
- adrian_b 3y ago"One to two orders of magnitude" may be true only for FP32 or FP16 and only when comparing a single GPU to a single CPU. For FP64, the ratio in the performance per watt between the few datacenter-only GPU models that support FP64 and CPUs has been only around 3 for multiple GPU and CPU generations. For inference using the AMX instruction set I have not seen any useful published benchmark that can show which is the ratio between the performance per watt of Emerald Rapids vs. NVIDIA, but the performance per dollar is certainly much worse for NVIDIA. What limits the performance is normally either the performance per dollar or the performance per watt, it seldom matters if the performance is provided by a single huge GPU or by multiple smaller GPUs or CPUs. While the top models of the server CPUs are overpriced, there are also models with very decent prices, like the 28-core Xeon 5512U @ $1230 or AMD Ryzen 9 7950X.
- ein0p 3y agoNobody uses FP64 for AI. Nobody uses consumer SKUs in the datacenters.