TensorFlow is indeed the one-eyed king here. We found that running with large SparseTensors, CPU training jobs is actually faster than on GPU nodes. An individu

TensorFlow is indeed the one-eyed king here. We found that running with large SparseTensors, CPU training jobs is actually faster than on GPU nodes. An individual job is faster on GPU, but SparseMatMul just blows up GPU memory use to the point that only one train job fits in its memory. SparseMatMul is inefficient on CPU as well, but you can keep getting some decent speedup by cranking up the parallel trade jobs per node. I did have great success using Netflix Vectorflow. I roughly gained 2x performance for my use case. It does leave some other performance on the table, no vectorized minibatches etc. If you are training on huge sparse datasets with a shallow ffw network, Vectorflow is a good alternative.