5 ms·

It's more on the framework that you use than nvidia at this point. Anything dockerized works with any compatible underlying hardware with no issues. Any optimiz

It's more on the framework that you use than nvidia at this point. Anything dockerized works with any compatible underlying hardware with no issues. Any optimization is again fragmented with FasterTransformer or TensorRT conversion with half baked layer supports which lags by 6months or more pretty much. NVAIE license is what nvidia wants enterprises to pay for using their bespoke cards in shared VRAM configuration by knee capping consumer cards which can very well do the same job better with more cuda cores but lesser memory. And don't even get me started on RIVA stack FP8 emulation is also never going to get backported instead only H100 & 4090s can make use of it

homarp 3y ago
NVAIE aka Nvidia AI enterprise, https://docs.nvidia.com/ai-enterprise/overview/0.1.0/platform-overview.html https://docs.nvidia.com/ai-enterprise/overview/0.1.0/platfor... RIVA: NVIDIA® Riva, a premium edition of NVIDIA AI Enterprise software, is a GPU-accelerated speech and translation AI SDK FasterTransformer: https://github.com/NVIDIA/FasterTransformer https://github.com/NVIDIA/FasterTransformer an highly optimized transformer-based encoder and decoder component, supported on pytorch, tensorflow and triton TensorRT, custom ml framework/ inference runtime from nvidia, https://developer.nvidia.com/tensorrt https://developer.nvidia.com/tensorrt, but you have to port your models
codethief 3y ago
> Any optimization is again fragmented with FasterTransformer or TensorRT conversion with half baked layer supports which lags by 6months or more pretty much. Thanks, I came here to see whether anything had changed since I last did ML stuff on Nvidia GPUs, and it looks like things are still the same.
- villgax 3y ago
  At this point the benefits of a GPU get outmatched by CPUs even if the latency is 5-10X since you can scale CPU cores cheaper than GPUs both on prem or on public cloud
  - kkielhofner 3y ago
    I'm not sure I agree with you. An RTX 4090 has over 16,000 cores and 1 TB/s of memory bandwidth. From what I understand (not really my thing) DDR5 tops out at 51 GB/s per module. CPUs and GPUs are so fundamentally different architecturally but for extremely parallel tasks GPUs are designed for CPU is very, very far behind. When I've done performance tests between CPU and GPU for my applications (speech) a $100 six year old GTX 1070 is 5x faster than a AMD Ryzen Threadripper PRO 5955WX[0] while consuming a fraction of the power and cost. If you look at the table the RTX 3090 and RTX 4090 are 17x and 27x respectively. The H100 benchmark of 12x is from a very early access benchmark with some driver and other issues. [0] - https://github.com/toverainc/willow-inference-server/tree/wisng#benchmarks https://github.com/toverainc/willow-inference-server/tree/wi...

homarp 3y ago

NVAIE aka Nvidia AI enterprise, https://docs.nvidia.com/ai-enterprise/overview/0.1.0/platform-overview.html https://docs.nvidia.com/ai-enterprise/overview/0.1.0/platfor... RIVA: NVIDIA® Riva, a premium edition of NVIDIA AI Enterprise software, is a GPU-accelerated speech and translation AI SDK FasterTransformer: https://github.com/NVIDIA/FasterTransformer https://github.com/NVIDIA/FasterTransformer an highly optimized transformer-based encoder and decoder component, supported on pytorch, tensorflow and triton TensorRT, custom ml framework/ inference runtime from nvidia, https://developer.nvidia.com/tensorrt https://developer.nvidia.com/tensorrt, but you have to port your models

codethief 3y ago

> Any optimization is again fragmented with FasterTransformer or TensorRT conversion with half baked layer supports which lags by 6months or more pretty much. Thanks, I came here to see whether anything had changed since I last did ML stuff on Nvidia GPUs, and it looks like things are still the same.

villgax 3y ago
At this point the benefits of a GPU get outmatched by CPUs even if the latency is 5-10X since you can scale CPU cores cheaper than GPUs both on prem or on public cloud
- kkielhofner 3y ago
  I'm not sure I agree with you. An RTX 4090 has over 16,000 cores and 1 TB/s of memory bandwidth. From what I understand (not really my thing) DDR5 tops out at 51 GB/s per module. CPUs and GPUs are so fundamentally different architecturally but for extremely parallel tasks GPUs are designed for CPU is very, very far behind. When I've done performance tests between CPU and GPU for my applications (speech) a $100 six year old GTX 1070 is 5x faster than a AMD Ryzen Threadripper PRO 5955WX[0] while consuming a fraction of the power and cost. If you look at the table the RTX 3090 and RTX 4090 are 17x and 27x respectively. The H100 benchmark of 12x is from a very early access benchmark with some driver and other issues. [0] - https://github.com/toverainc/willow-inference-server/tree/wisng#benchmarks https://github.com/toverainc/willow-inference-server/tree/wi...