Yeah but the challenge lately has been that efficiency gains have also made the capabilities of large compute clusters even more powerful. Transformers were int

Yeah but the challenge lately has been that efficiency gains have also made the capabilities of large compute clusters even more powerful. Transformers were introduced as an efficient alternative to RNNs and conv nets, then it turned out that however much they supercharged a single GPU, they were dramatically more powerful deployed at datacenter scale.