Great point! If I recall correctly, this team (well, nearly all the top teams from DawnBench) took Page's code and wrestled it into the multi-GPU realm. I'm a s

Great point! If I recall correctly, this team (well, nearly all the top teams from DawnBench) took Page's code and wrestled it into the multi-GPU realm. I'm a sucker for simplicity, as much as is reasonable (this codebase currently does not use JIT or any custom kernels! (!!!)), and also making sure that the average practitioner (like me) could do something workable without having to pay tons of money. My computing costs are $50 a month currently, i.e. the cost of Pro Colab. And we were able to break the single-GPU WR, and we're really close to pushing past any of the official multi-GPU submissions (old as they may be!). I took David's work in a different direction and just kept it true I think to the original spirit of things. Cycle times for experimentation are king in ML when it comes to the speed of research progress, regardless of what anyone else might tell you. Having tons of hardware may be really flashy and useful for the end product, but it's certainly not needed for much of the lo-fi, day-to-day stuff. That said, the A100 is definitely a step up. It is under 2x, though, as we are basically only memory-and-slow-backprop-kernel limited now, not as much by the convolutions (which now are among the shorter operations). Running https://github.com/99991/cifar10-fast-simple https://github.com/99991/cifar10-fast-simple on my end gave me 17.2 seconds, vs the 24 seconds that Dave reported on the V100 (though the lovely author of that repo, @99991, was able to get faster speeds on their personal A100 setup). So we're definitely in that weird regime where moving everything to massively scaled matrix multiplies when possible is preferred, and sometimes that's...tricky for a few of these operations.