According to: https://github.com/soumith/convnet-benchmarks https://github.com/soumith/convnet-benchmarks, it's an NVIDIA TitanX (12GB GPU memory), which is pre

According to: https://github.com/soumith/convnet-benchmarks https://github.com/soumith/convnet-benchmarks, it's an NVIDIA TitanX (12GB GPU memory), which is pretty much the top of the line GPU for training neural nets. From my limited experience, most deep learning in research - including much of the state of the art - is done on single GPUs. (My guess is that if your model doesn't fit in 12GB, your model has way too many parameters to practically train anyway).