Another thing to note is that you can in theory train with half precision without sacrificing much accuracy. There isn't that much support for it yet, for AMD a

Another thing to note is that you can in theory train with half precision without sacrificing much accuracy. There isn't that much support for it yet, for AMD anyway that I've seen, but Vega 64 has 2x speed half precision (25 TFLOPS) on paper. Out of Nvidia cards only Tesla and Volta have 2x (or more) speed mixed precision ops (in Voltas case actually much faster because of the Tensor Cores) but they are in a very different price bracket.