3 ms·
> Any optimization is again fragmented with FasterTransformer or TensorRT conversion with half baked layer supports which lags by 6months or more pretty much.
by codethief 3y ago
> Any optimization is again fragmented with FasterTransformer or TensorRT conversion with half baked layer supports which lags by 6months or more pretty much.
Thanks, I came here to see whether anything had changed since I last did ML stuff on Nvidia GPUs, and it looks like things are still the same.
- villgax 3y agoAt this point the benefits of a GPU get outmatched by CPUs even if the latency is 5-10X since you can scale CPU cores cheaper than GPUs both on prem or on public cloud
- kkielhofner 3y agoI'm not sure I agree with you. An RTX 4090 has over 16,000 cores and 1 TB/s of memory bandwidth. From what I understand (not really my thing) DDR5 tops out at 51 GB/s per module. CPUs and GPUs are so fundamentally different architecturally but for extremely parallel tasks GPUs are designed for CPU is very, very far behind. When I've done performance tests between CPU and GPU for my applications (speech) a $100 six year old GTX 1070 is 5x faster than a AMD Ryzen Threadripper PRO 5955WX[0] while consuming a fraction of the power and cost. If you look at the table the RTX 3090 and RTX 4090 are 17x and 27x respectively. The H100 benchmark of 12x is from a very early access benchmark with some driver and other issues. [0] - https://github.com/toverainc/willow-inference-server/tree/wisng#benchmarks https://github.com/toverainc/willow-inference-server/tree/wi...