4 ms·
That's exactly what Nvidia is doing with tensor cores.
by fooker 11mo ago
That's exactly what Nvidia is doing with tensor cores.
- bjourne 11mo agoExcept the native width of Tensor Cores are about 8-32 (depending on scalar type), whereas the width of TPUs is up to 256. The difference in scale is massive.
- saagarjha 11mo agoNobody cares about width; they care about TFLOPs.
- fooker 11mo agoIf it turns out to be useful, Nvidia can't just tweak a parameter in their verilog and declare victory? If not, what's fundamentally difficult about doing 32 vs 256 here?
- neilmovva 11mo agoI think Hopper's native matmul tile is 64x64, and Blackwell is 128x128. see this blog for a reference on Blackwell: https://hazyresearch.stanford.edu/blog/2025-03-15-tk-blackwell https://hazyresearch.stanford.edu/blog/2025-03-15-tk-blackwe...