Correct, GPUs can do SGD iterations much much faster once the data is in GPU RAM. But if you have a ton of data relative to compute, you might be better off wi

Correct, GPUs can do SGD iterations much much faster once the data is in GPU RAM. But if you have a ton of data relative to compute, you might be better off with a shallower model and/or fewer iterations. Here’s one start-up serving these sorts of projects: https://www.memverge.com/ https://www.memverge.com/ When GPUs kickstarted deep learning research in 2012, people had already studied shallow models on mapreduce for a decade or so. Once NVME / Optane and modern CPUs get 1-10TB of useful “memory” in the hands of grad students, there should be another wave of new research. To date, my experience has been that 1TB of “memory” is only commonly available in industry.