If anyone is curious, https://www.gwern.net/docs/ai/2021-jouppi.pdf#page=5 https://www.gwern.net/docs/ai/2021-jouppi.pdf#page=5 provides a breakdown of TPU work

If anyone is curious, https://www.gwern.net/docs/ai/2021-jouppi.pdf#page=5 https://www.gwern.net/docs/ai/2021-jouppi.pdf#page=5 provides a breakdown of TPU workload by type. I disagree with Hooker's hardware lottery thesis for the simple reason that if it was true, she would be able to point to examples of DL-competitive methods which use the same amount of FLOPS to get much superior results (because everyone is so badly neglecting them due to the 'lottery'), or at least show better scaling curves so that they will at some point surpass DL, but at much worse wallclock due to whatever hardware specializations favor DL and penalize those alternatives. This is how DL operated: they ran on CPUs, very slowly compared to GPUs, but they did run, enabling eventual exploitation of GPUs. And that is how they could start a virtuous circle: the success of DL, because it's the right thing, starting from hardware not even remotely designed for DL (like 2010-era GPUs were not) and designed to favor other tasks (classic GPGU stuff) has pulled hardware in its wake. So, what are the DL-like things running on CPU or contemporary GPUs? The only actual example Hooker gives is capsule networks - which I thought were doomed when they were unveiled, a poor attempt at stuff soft attention did better already, and have not impressed anyone in the 5 years since, particularly as larger NNs (whether CNN, Transformer, or MLP) continue to deliver what capsnets promised. Nor have any more compelling examples arisen in the years since. With no examples or comparisons, it boils down to nothing much.