The connection is unfortunately crucial – apart from deep learning, there aren't actually that many problems that require the amount of computation to justify t

The connection is unfortunately crucial – apart from deep learning, there aren't actually that many problems that require the amount of computation to justify the transfer. BUT: considering the CPU is really just a very expensive traffic conductor in the way these frameworks work, an option may be to combine GPUs with fast storage (SSD) and/or maybe a sort of second-level RAM. Scaling across GPUs is pretty easy. You can run batches in parallel and combine the results, and only the learned parameters need to be transferred once you have distributed the data.