4 ms·

In a sense, GPUs are only great at matrix-matrix multiplication. For anything else you would only get 7% of the FLOPs/s compared to it(989 vs 67 TFLOP/s for H10

In a sense, GPUs are only great at matrix-matrix multiplication. For anything else you would only get 7% of the FLOPs/s compared to it(989 vs 67 TFLOP/s for H100)[1]. [1]: https://www.nvidia.com/en-in/data-center/h100/ https://www.nvidia.com/en-in/data-center/h100/

winwang 1y ago
lol, I haven't thought about it like that, true. though of course, I mean compared to CPUs :P I try and use tensor cores for non-obvious things every now and then. The most promising so far seems to be for linear arithmetic in Datalog, but that's just matrix-vector/gemv
- harperlee 1y ago
  Could you expand the Datalog example? I'm quite interested
  - winwang 1y ago
    This was just a brief moment of thought over a year ago, but I can try to summarize. I was thinking about how to unify variables in certain simple Datalog settings. If we think of a clause as a vector of variables, then simple unifications can look like just a gather operation. A gather can be thought of as a matrix-vector multiplication, but that's not really useful (performance wise). But if those variables are also in a linear equation, then it becomes possibly-useful, e.g. for something like `P(x, y) :- E(x, 3x+4y)`
cma 1y ago
That link says "* With sparsity". For extremely sparse matrixes you can get more than 989 TFLOPS on CPU, if we're counting elided operations in TFLOPS.
- YetAnotherNick 1y ago
  I am counting FP16/BF16 without sparsity, which is used in majority of AI.
  - cma 1y ago
    That change checks out then. They didn't see much need for FP16 outside of that so no longer run it at double FP32 rate outside of tensor cores (unless I'm mixing that up with AMD). Other forms of sparsity are heavily used at training time now, like block compression in Deepseek.
deleted 1y ago
[deleted]
randomgermanguy 1y ago
Funnily, they're far from being optimal for GEMM ops (especially in terms of power consumption). For GEMM you need to visit each row/vec n-times so theres a bunch of data-reuse going on, which isn't optimal for GPUs since you can't keep that all so close to your processing-units. And while the tensor-cores kinda implement this i think they don't quite scale up to a full sized systolic array, which is you would want for larger matrix multiplications. Also just a simpler view: with GPUs most of their silicon is spent NOT tensor-core, so just from that you know its not optimal i guess. Just referring to that FLOP/s number doesn't really mean much nowadays with tensor-cores and sparsity. In my eyes the big win of GPUs are that not only are they pretty good at GEMMs but also really good at a lot of other easily parallelizable tasks PLUS they're comparatively easy to program ^^

winwang 1y ago

lol, I haven't thought about it like that, true. though of course, I mean compared to CPUs :P I try and use tensor cores for non-obvious things every now and then. The most promising so far seems to be for linear arithmetic in Datalog, but that's just matrix-vector/gemv

harperlee 1y ago
Could you expand the Datalog example? I'm quite interested
- winwang 1y ago
  This was just a brief moment of thought over a year ago, but I can try to summarize. I was thinking about how to unify variables in certain simple Datalog settings. If we think of a clause as a vector of variables, then simple unifications can look like just a gather operation. A gather can be thought of as a matrix-vector multiplication, but that's not really useful (performance wise). But if those variables are also in a linear equation, then it becomes possibly-useful, e.g. for something like `P(x, y) :- E(x, 3x+4y)`

cma 1y ago

That link says "* With sparsity". For extremely sparse matrixes you can get more than 989 TFLOPS on CPU, if we're counting elided operations in TFLOPS.

YetAnotherNick 1y ago
I am counting FP16/BF16 without sparsity, which is used in majority of AI.
- cma 1y ago
  That change checks out then. They didn't see much need for FP16 outside of that so no longer run it at double FP32 rate outside of tensor cores (unless I'm mixing that up with AMD). Other forms of sparsity are heavily used at training time now, like block compression in Deepseek.

deleted 1y ago

[deleted]

randomgermanguy 1y ago

Funnily, they're far from being optimal for GEMM ops (especially in terms of power consumption). For GEMM you need to visit each row/vec n-times so theres a bunch of data-reuse going on, which isn't optimal for GPUs since you can't keep that all so close to your processing-units. And while the tensor-cores kinda implement this i think they don't quite scale up to a full sized systolic array, which is you would want for larger matrix multiplications. Also just a simpler view: with GPUs most of their silicon is spent NOT tensor-core, so just from that you know its not optimal i guess. Just referring to that FLOP/s number doesn't really mean much nowadays with tensor-cores and sparsity. In my eyes the big win of GPUs are that not only are they pretty good at GEMMs but also really good at a lot of other easily parallelizable tasks PLUS they're comparatively easy to program ^^