My apologies! Titles are hard. Indeed, this is training to 94% accuracy in only ~12.3-~12.4 seconds on a single GPU (so human-affordable), which gets us closer

My apologies! Titles are hard. Indeed, this is training to 94% accuracy in only ~12.3-~12.4 seconds on a single GPU (so human-affordable), which gets us closer to massive scaled testing for neural network phenomenology. Previously a number of years ago (maybe 4-5), the world record was ~10 minutes, so quite a lot of increase! I've picked up the torch as nearly everyone has moved on from it, but finishing a near-optimal version of this benchmark will transfer to transformers and whatever training techniques come after them in the future. I hope to do similarly for language models in the future, but writing it from scratch (I've only started) is a haul. But this is also an entirely hackable codebase, so you can change anything from the dataloader to the final SGD step call within a few minutes (without having to do several minutes - hours of debugging or rewriting). I really appreciate you linking the CIFAR10 explanation.