Physical space on the motherboard IMO is the true limiting factor here. And this is what made Google's TPU interesting IMO. I don't think it demonstrated worl

Physical space on the motherboard IMO is the true limiting factor here. And this is what made Google's TPU interesting IMO. I don't think it demonstrated world-beating performance, but it likely fit a strict power and size envelope as an add-on board in the their datacenters, hence all the bragging about perf/W without a single absolute performance benchmark.