Are TPUs too application specific to replace GPUs? It seems cloud TPUs could be competitive with GPUs in terms of $ per number of target epochs, provided you ca

Are TPUs too application specific to replace GPUs? It seems cloud TPUs could be competitive with GPUs in terms of $ per number of target epochs, provided you can do data parallelism for your workloads. Also, IBM offers bare metal pricing which is somewhat cheaper than virtualized instances attached to GPUs (and faster too). I think GPU virtualization is not quite there yet because Nvidia does not give access to core GPU functionality needed for efficient virtualization - you're stuck with using their closed-source libraries and drivers.