3 ms·

The RTX 3090 has 35.58 TFlops FP32 performance, or 285.48 FP16 according to https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units#GeForce_30_se

The RTX 3090 has 35.58 TFlops FP32 performance, or 285.48 FP16 according to https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units#GeForce_30_series https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_proces... EDIT: I fell for NVIDIA's marketing. The dense FP16 performance is only half of 284.48, which is 142. Thanks to adgjlsfhk1 for the correction.

adgjlsfhk1 4y ago
That 285 is listed as (2:1 sparse) which means it's only valid for matrices where 2 out of every 4 numbers are zero. For dense matrices it's half that.
- bee_rider 4y ago
  Are 2:1 sparse matrices a common thing? It seems weird, like clearly that’s not sparse enough to want to use, like, sparse matrix “CSR” style storage or something, haha. I would just treat it as dense I guess.
  - adgjlsfhk1 4y ago
    They aren't. As far as I can tell, Nvidia does this to be able to double the number of TFlops they put on their website. (this might be a little unfair, the real reason is that in ML it might be possible to train a NN such that your matrices have this structure, but I haven't seen anyone other than Nvidia use it)
    - bee_rider 4y ago
      I’m trying to think of cases where it might accidentally come up, and all I can think of is something like “oops I used complex but my values are actually real.”
    - Firadeoclus 4y ago
      What you might do is train using dense matrices, then sparsify those (pick the 2 out of each set of 4 weights that are closest to zero, mask them out), then do a few more training iterations with the mask in place. It turns out that even without the extra training iterations you often lose surprisingly little in terms of quality of output. In reality you can sparsify a lot more, but 2 out of 4 is so simple and easy to implement in hardware, more complex schemes are much harder to justify. However, small matmuls (say, <2048 bytes in the K dimension) won't get anywhere near 2x performance.
  - dotnet00 4y ago
    There has been some work in that direction but it hasn't really caught on as fast as NVIDIA may have expected it to.
    - lostmsu 4y ago
      Yeah, still waiting for this feature to be available in PyTorch natively.
  - deleted 4y ago
    [deleted]

adgjlsfhk1 4y ago

That 285 is listed as (2:1 sparse) which means it's only valid for matrices where 2 out of every 4 numbers are zero. For dense matrices it's half that.

bee_rider 4y ago
Are 2:1 sparse matrices a common thing? It seems weird, like clearly that’s not sparse enough to want to use, like, sparse matrix “CSR” style storage or something, haha. I would just treat it as dense I guess.
- adgjlsfhk1 4y ago
  They aren't. As far as I can tell, Nvidia does this to be able to double the number of TFlops they put on their website. (this might be a little unfair, the real reason is that in ML it might be possible to train a NN such that your matrices have this structure, but I haven't seen anyone other than Nvidia use it)
  - bee_rider 4y ago
    I’m trying to think of cases where it might accidentally come up, and all I can think of is something like “oops I used complex but my values are actually real.”
  - Firadeoclus 4y ago
    What you might do is train using dense matrices, then sparsify those (pick the 2 out of each set of 4 weights that are closest to zero, mask them out), then do a few more training iterations with the mask in place. It turns out that even without the extra training iterations you often lose surprisingly little in terms of quality of output. In reality you can sparsify a lot more, but 2 out of 4 is so simple and easy to implement in hardware, more complex schemes are much harder to justify. However, small matmuls (say, <2048 bytes in the K dimension) won't get anywhere near 2x performance.
- dotnet00 4y ago
  There has been some work in that direction but it hasn't really caught on as fast as NVIDIA may have expected it to.
  - lostmsu 4y ago
    Yeah, still waiting for this feature to be available in PyTorch natively.
- deleted 4y ago
  [deleted]