Also, making neural networks faster/cheaper is a big part of how they advance. We've known about neural architectures since the 70s, but we couldn't build them

Also, making neural networks faster/cheaper is a big part of how they advance. We've known about neural architectures since the 70s, but we couldn't build them big enough to be actually useful until the advent of the GPU. Similarly, the LLM breakthrough was because someone decided it was worth spending millions of dollars to train one. Efficiency improvements lower that barrier for all future development (or alternatively, allow us to build even bigger models for the same cost.)