4 ms·
Except the native width of Tensor Cores are about 8-32 (depending on scalar type), whereas the width of TPUs is up to 256. The difference in scale is massive.
by bjourne 11mo ago
Except the native width of Tensor Cores are about 8-32 (depending on scalar type), whereas the width of TPUs is up to 256. The difference in scale is massive.
- saagarjha 11mo agoNobody cares about width; they care about TFLOPs.
- fooker 11mo agoIf it turns out to be useful, Nvidia can't just tweak a parameter in their verilog and declare victory? If not, what's fundamentally difficult about doing 32 vs 256 here?
- neilmovva 11mo agoI think Hopper's native matmul tile is 64x64, and Blackwell is 128x128. see this blog for a reference on Blackwell: https://hazyresearch.stanford.edu/blog/2025-03-15-tk-blackwell https://hazyresearch.stanford.edu/blog/2025-03-15-tk-blackwe...