3 ms·
At this point the benefits of a GPU get outmatched by CPUs even if the latency is 5-10X since you can scale CPU cores cheaper than GPUs both on prem or on publi
by villgax 3y ago
At this point the benefits of a GPU get outmatched by CPUs even if the latency is 5-10X since you can scale CPU cores cheaper than GPUs both on prem or on public cloud
- kkielhofner 3y agoI'm not sure I agree with you. An RTX 4090 has over 16,000 cores and 1 TB/s of memory bandwidth. From what I understand (not really my thing) DDR5 tops out at 51 GB/s per module. CPUs and GPUs are so fundamentally different architecturally but for extremely parallel tasks GPUs are designed for CPU is very, very far behind. When I've done performance tests between CPU and GPU for my applications (speech) a $100 six year old GTX 1070 is 5x faster than a AMD Ryzen Threadripper PRO 5955WX[0] while consuming a fraction of the power and cost. If you look at the table the RTX 3090 and RTX 4090 are 17x and 27x respectively. The H100 benchmark of 12x is from a very early access benchmark with some driver and other issues. [0] - https://github.com/toverainc/willow-inference-server/tree/wisng#benchmarks https://github.com/toverainc/willow-inference-server/tree/wi...