We have been using GPUs with GKE for a while. At some point, we used 20+ GPUs in an production workflow without any problems. Everything generally works well,

We have been using GPUs with GKE for a while. At some point, we used 20+ GPUs in an production workflow without any problems. Everything generally works well, maybe except the initial phase when some containers won't port well from nvidia-docker-compose due to problems with Cuda libraries. Ideally, you need to match the version of Cuda everywhere. My dev setup for quick experimentation with GPU docker container on GKE: https://tensorflight.blog/2018/02/23/dev-environment-for-gke/ https://tensorflight.blog/2018/02/23/dev-environment-for-gke... .