Well, you lose some performance to distributed memory overhead of course, plus the TPU measure for that number is a previous generation TPU. The new ones are a

Well, you lose some performance to distributed memory overhead of course, plus the TPU measure for that number is a previous generation TPU. The new ones are a decent bit faster. Plus you get four chips per "Cloud TPU". Putting all that together, TPUs are certainly a very competitive offering.