Cherry-picking a few papers doesn‘t tell anything. If at all it shows what people have achieved who pushed the envelope to the extreme, mostly at Google where p

Cherry-picking a few papers doesn‘t tell anything. If at all it shows what people have achieved who pushed the envelope to the extreme, mostly at Google where people can afford to not care about cost. 99.9% of the work is done using small numbers of GPUs, and that hasn‘t changed much in recent years, except for the improvements in GPU architectures. Draw this graph and you get a very different story.