At least back in 2012, Google research seemed to suggest that distributed CPU training of large models could sometimes be preferable to fitting within the limit

At least back in 2012, Google research seemed to suggest that distributed CPU training of large models could sometimes be preferable to fitting within the limits of GPUs: http://research.google.com/archive/large_deep_networks_nips2012.html http://research.google.com/archive/large_deep_networks_nips2...