It's not soooo appalling once you dig further into what's happening. There's been a tug-of-war between compute bottlenecks and memory bottlenecks in accelerato

It's not soooo appalling once you dig further into what's happening. There's been a tug-of-war between compute bottlenecks and memory bottlenecks in accelerators for quite a while. Big increases in FLOPs requires faster memory access, so you tend to get hardware releases alternating between emphasizing compute gains and bus bandwidth gains. Data I/O is just the next layer out... once your base training is running fast enough, it becomes progressively harder to keep the model fed with data.