3 ms·
Right. As I understand it, training on gpus isn’t limited by the speed of matrix multiplications. It’s limited by memory bandwidth. So a faster ASIC for matrix
by josephg 3y ago
Right. As I understand it, training on gpus isn’t limited by the speed of matrix multiplications. It’s limited by memory bandwidth. So a faster ASIC for matrix operations won’t help. It’ll just sit idle while the system stalls waiting for data to become available.
That’s why having 96gb (with another 480gb or whatever) available via high speed interconnect is a big deal. It means we can train bigger models faster.