2 ms·

> so you had to have lots of operand re-use to not be memory-bound Looking at Nvidia's spec sheet, an H100 SXM can do 989 tf32 teraflops (or 67 non-tensor core

> so you had to have lots of operand re-use to not be memory-bound Looking at Nvidia's spec sheet, an H100 SXM can do 989 tf32 teraflops (or 67 non-tensor core fp32 teraflops?) and 3.35 TB/s memory (HBM) bandwidth, so ... similar problem?

pklausler 2y ago
There is caching today.
- ryao 2y ago
  The cache hitrate is effectively 0 for LLMs since the datasets are so huge.