3 ms·

One interesting consequence of the model consolidation happening in the LLM world is that it's actually easier for AMD's hardware to be competitive. It used to

One interesting consequence of the model consolidation happening in the LLM world is that it's actually easier for AMD's hardware to be competitive. It used to be that you had to support a ton of ops since nobody is going to use your GPU for just a single model. Nowadays, most models are just transformers with only a few operations inside, albeit repeated many times over. On top of that, you can build a reasonable application or business using only one or two LLMs (i.e. LLaMA 2 + Whisper, etc). So as long as they can get those running fast, they can sell GPUs to people who don't need the flexibility and breadth of CUDA.

KRAKRISMOTT 3y ago
The mighty dot product and matrix multiplication strikes again!
oivey 3y ago
I’m not sure this is true. Nvidia ships hardware today that is specialized for transformers, and presumably AMD needs something similar on their side. An arms race of specific hardware like that is probably not to their advantage. It makes the interoperability story even harder. They still need a usable GPGPU language to glue all that hardware together, too.
- dhruvdh 3y ago
  There is nothing special about Nvidia hardware outside of them having significantly better lower precision FLOPs before anyone else. Their marketing chose to call this a "transformer engine" but there is no actual "engine" in silicon. Like AMD announced today, and is shown in the article we're commenting on, AMD now is 1.3 times faster at FP8, INT8 flops with and without sparsity, meaning their "transformer engine" is 1.3 times faster. The rest of your comment doesn't make sense. See the other comments on software in this thread.
ilaksh 3y ago
The idea of model architecture making fast hardware design easier is what makes https://github.com/tinygrad/tinygrad https://github.com/tinygrad/tinygrad so interesting.
- dharma1 3y ago
  yep! I think geohot could singlehandedly turn AMD into a winner if (big if) they get tinygrad competitive perf wise on AMD hw. So far it's not

KRAKRISMOTT 3y ago

The mighty dot product and matrix multiplication strikes again!

oivey 3y ago

I’m not sure this is true. Nvidia ships hardware today that is specialized for transformers, and presumably AMD needs something similar on their side. An arms race of specific hardware like that is probably not to their advantage. It makes the interoperability story even harder. They still need a usable GPGPU language to glue all that hardware together, too.

dhruvdh 3y ago
There is nothing special about Nvidia hardware outside of them having significantly better lower precision FLOPs before anyone else. Their marketing chose to call this a "transformer engine" but there is no actual "engine" in silicon. Like AMD announced today, and is shown in the article we're commenting on, AMD now is 1.3 times faster at FP8, INT8 flops with and without sparsity, meaning their "transformer engine" is 1.3 times faster. The rest of your comment doesn't make sense. See the other comments on software in this thread.

ilaksh 3y ago

The idea of model architecture making fast hardware design easier is what makes https://github.com/tinygrad/tinygrad https://github.com/tinygrad/tinygrad so interesting.

dharma1 3y ago
yep! I think geohot could singlehandedly turn AMD into a winner if (big if) they get tinygrad competitive perf wise on AMD hw. So far it's not