4 ms·

> Can you link to benchmarks or other analysis supporting this claim? Current generation nVidia: https://en.wikipedia.org/wiki/GeForce_40_series#Desktop https:

> Can you link to benchmarks or other analysis supporting this claim? Current generation nVidia: https://en.wikipedia.org/wiki/GeForce_40_series#Desktop https://en.wikipedia.org/wiki/GeForce_40_series#Desktop Current generation AMD: https://en.wikipedia.org/wiki/Template:AMD_Radeon_RX_7000 https://en.wikipedia.org/wiki/Template:AMD_Radeon_RX_7000 The key performance characteristics are processing power TFlops, and memory bandwidth GB/s. nVidia 4080 which costs $1200: 43 TFlops FP16, 43 TFlops FP32, 0.672 TFlops FP64, 717 GB/s memory. AMD 7900 XTX which costs $1000: 93 TFlops FP16, 47 TFlops FP32, 1.5 TFlops FP64, 960 GB/s memory. Note that for applications which bottleneck on FP16 compute (many ML workloads) or FP64 compute (many traditional HPC workloads: numerical solvers, fluid dynamics, etc), the 7900 XTX even outperforms the 4090 which costs $1600.

ColonelPhantom 3y ago
In ML workloads, usually the FP16 operations are matrix operations. On RDNA3, these execute at the same rate as normal shader/vector operations, but on Nvidia RTX cards there are Tensor cores which accelerate them. The Ada whitepaper lists 48.7 shader TFlops (not 43 because boost vs base clock), and 195 TFlops for FP16 Tensor with FP16 Accumulate. That's 4 times faster than regular, and almost double what the XTX lists! Ampere and newer also have native sparsity support which means that you can skip over the zeroes 'for free', which Nvidia uses to market double the TFlops, which is kind of misleading imo. But the 195 TFlops are even before sparsity is included! I'm not sure if the 93 TFlops (120 with boost clocks) on AMD are with FP16 or FP32 accumulation, as with FP32 accumulation the 4080 slows down significantly and gets much closer with 97.5 TFlops. Intel Xe-HPG (used in the Arc A cards) also offers very aggressive matrix acceleration via XMX, with 137.6 FP16 TFlops at base clock, vs. 17.2 FP32 TFlops.
- Const-me 3y ago
  You’re comparing general-purpose computations to a proprietary feature with limited applicability. For example, in my Mistral implementation the most expensive compute shader is handling matrices compressed with a custom codec unknown to tensor cores. This is not an apples-to-apples comparison. We use FLOPS numbers to compare processors of different architectures because they’re the same FLOPS everywhere.
  - ColonelPhantom 3y ago
    >This is not an apples-to-apples comparison. Neither is FLOPS. RDNA3 FLOPS numbers are greatly inflated, because they are only applicable to fairly specific VOPD and WMMA instructions, the former being for two MADs at the same time, and the latter being applicable in the same cases as tensor cores. Besides, it should be possible to use the tensor cores combined with the codec: you can use vector hardware to decode matrix values to fp16/fp32, then yeet those into the tensor cores. Although most of the time will probably be spent on the decoding part, assuming you're doing matrix-vector multiplication and not matrix-matrix (which might be different with MoE models like Mixtral 8x7B?)
    - Const-me 3y ago
      RDNA3 FLOPS numbers are real. Shader compiler happily uses VOPD for quite a few things including FP32 FMA. An independent researcher measured 62.9 FP32 TFlops (boost) in Vulkan on 7900XTX, see “Full GPU Throughput – Vulkan” section on that article: https://chipsandcheese.com/2023/01/07/microbenchmarking-amds-rdna-3-graphics-architecture/ https://chipsandcheese.com/2023/01/07/microbenchmarking-amds...
einpoklum 3y ago
That is interesting, even though I would need to look at non-consumer-GFX cards. What about actual benchmarks, though? Passmark, for example: https://www.videocardbenchmark.net/directCompute.html https://www.videocardbenchmark.net/directCompute.html has: GeForce RTX 4090 28,916 RTX 6000 Ada 23,296 GeForce RTX 4080 22,016 Radeon RX 7900 XTX 18,767 and Geekbench 6: https://browser.geekbench.com/opencl-benchmarks https://browser.geekbench.com/opencl-benchmarks has: GeForce RTX 4090 321928 L40 292357 <- L40S is out and may be better RTX 6000 Ada 274348 H100 267514 NVIDIA GeForce RTX 239863 Radeon RX 7900 XTX 199412
- NavinF 3y ago
  Neither of those benchmarks is relevant to ML which is what GP is talking about
  - izacus 3y ago
    So what is relevant? As other poster said, nVidia tends to outpace AMD cards in pretty much every use case in real life.
    - Const-me 3y ago
      The only important numbers are processing power TFlops, and memory bandwidth GB/s. If your compute shader doesn’t approach the theoretical limit of either computations or memory, it doesn’t mean there’s anything wrong with the GPU. Here’s incomplete list of possible reasons. ● Insufficient parallelism of the problem. Some problems are inherently sequential. ● Poor HLSL programming skills. For example, a compute shaders with 32 threads/group wastes 50% of compute units of most AMD GPUs, the correct number for AMD is 64 threads/group, or a multiple of 64. BTW, nVidia and Intel are fine with 64 threads/group, they run 1 thread group as 2 wavefronts which does not waste any resources. ● The problem being too small to compensate for the overhead. For example, CPUs multiply two 4x4 matrices in a small fraction of a time it takes to dispatch a compute shader for that. You gonna need much larger matrices for GPGPU to win.

ColonelPhantom 3y ago

In ML workloads, usually the FP16 operations are matrix operations. On RDNA3, these execute at the same rate as normal shader/vector operations, but on Nvidia RTX cards there are Tensor cores which accelerate them. The Ada whitepaper lists 48.7 shader TFlops (not 43 because boost vs base clock), and 195 TFlops for FP16 Tensor with FP16 Accumulate. That's 4 times faster than regular, and almost double what the XTX lists! Ampere and newer also have native sparsity support which means that you can skip over the zeroes 'for free', which Nvidia uses to market double the TFlops, which is kind of misleading imo. But the 195 TFlops are even before sparsity is included! I'm not sure if the 93 TFlops (120 with boost clocks) on AMD are with FP16 or FP32 accumulation, as with FP32 accumulation the 4080 slows down significantly and gets much closer with 97.5 TFlops. Intel Xe-HPG (used in the Arc A cards) also offers very aggressive matrix acceleration via XMX, with 137.6 FP16 TFlops at base clock, vs. 17.2 FP32 TFlops.

Const-me 3y ago
You’re comparing general-purpose computations to a proprietary feature with limited applicability. For example, in my Mistral implementation the most expensive compute shader is handling matrices compressed with a custom codec unknown to tensor cores. This is not an apples-to-apples comparison. We use FLOPS numbers to compare processors of different architectures because they’re the same FLOPS everywhere.
- ColonelPhantom 3y ago
  >This is not an apples-to-apples comparison. Neither is FLOPS. RDNA3 FLOPS numbers are greatly inflated, because they are only applicable to fairly specific VOPD and WMMA instructions, the former being for two MADs at the same time, and the latter being applicable in the same cases as tensor cores. Besides, it should be possible to use the tensor cores combined with the codec: you can use vector hardware to decode matrix values to fp16/fp32, then yeet those into the tensor cores. Although most of the time will probably be spent on the decoding part, assuming you're doing matrix-vector multiplication and not matrix-matrix (which might be different with MoE models like Mixtral 8x7B?)
  - Const-me 3y ago
    RDNA3 FLOPS numbers are real. Shader compiler happily uses VOPD for quite a few things including FP32 FMA. An independent researcher measured 62.9 FP32 TFlops (boost) in Vulkan on 7900XTX, see “Full GPU Throughput – Vulkan” section on that article: https://chipsandcheese.com/2023/01/07/microbenchmarking-amds-rdna-3-graphics-architecture/ https://chipsandcheese.com/2023/01/07/microbenchmarking-amds...

einpoklum 3y ago

That is interesting, even though I would need to look at non-consumer-GFX cards. What about actual benchmarks, though? Passmark, for example: https://www.videocardbenchmark.net/directCompute.html https://www.videocardbenchmark.net/directCompute.html has: GeForce RTX 4090 28,916 RTX 6000 Ada 23,296 GeForce RTX 4080 22,016 Radeon RX 7900 XTX 18,767 and Geekbench 6: https://browser.geekbench.com/opencl-benchmarks https://browser.geekbench.com/opencl-benchmarks has: GeForce RTX 4090 321928 L40 292357 <- L40S is out and may be better RTX 6000 Ada 274348 H100 267514 NVIDIA GeForce RTX 239863 Radeon RX 7900 XTX 199412

NavinF 3y ago
Neither of those benchmarks is relevant to ML which is what GP is talking about
- izacus 3y ago
  So what is relevant? As other poster said, nVidia tends to outpace AMD cards in pretty much every use case in real life.
  - Const-me 3y ago
    The only important numbers are processing power TFlops, and memory bandwidth GB/s. If your compute shader doesn’t approach the theoretical limit of either computations or memory, it doesn’t mean there’s anything wrong with the GPU. Here’s incomplete list of possible reasons. ● Insufficient parallelism of the problem. Some problems are inherently sequential. ● Poor HLSL programming skills. For example, a compute shaders with 32 threads/group wastes 50% of compute units of most AMD GPUs, the correct number for AMD is 64 threads/group, or a multiple of 64. BTW, nVidia and Intel are fine with 64 threads/group, they run 1 thread group as 2 wavefronts which does not waste any resources. ● The problem being too small to compensate for the overhead. For example, CPUs multiply two 4x4 matrices in a small fraction of a time it takes to dispatch a compute shader for that. You gonna need much larger matrices for GPGPU to win.