Since Ersatz got mentioned, I'd like to point out that we don't do true distributed GPU yet (it's planned). Most of the models support a single GPU with the RNN

Since Ersatz got mentioned, I'd like to point out that we don't do true distributed GPU yet (it's planned). Most of the models support a single GPU with the RNN models supporting multiple GPUs on the same machine. Of course, that's still way way faster than your laptop. Distributing computation across GPUs has been addressed here: http://techblog.netflix.com/2014/02/distributed-neural-networks-with-gpus.html http://techblog.netflix.com/2014/02/distributed-neural-netwo... We looked at a similar scheme, but didn't see it as practical on something like AWS Also here: http://www.stanford.edu/~acoates/papers/CoatesHuvalWangWuNgCatanzaro_icml2013.pdf http://www.stanford.edu/~acoates/papers/CoatesHuvalWangWuNgC... In this case, they are using infiniband, a high speed ethernet card/bus/whatever-you-call-it, in order to share data very quickly between multiple systems. This is basically like google's experiment but w/ GPUs (and thus way cheaper?) I'll let Adam speak to his strategy. It's an open research question but that doesn't mean there aren't any options or that those options are even terribly difficult given the right resources. Of course, real world performance is another story... whether you need and/or can use all that capacity, etc.