The short answer is yes, training neural networks it is compute bound in the sense that substantially improving computational throughput would enable bigger mod

The short answer is yes, training neural networks it is compute bound in the sense that substantially improving computational throughput would enable bigger models to be trained on bigger datasets than is currently possible. The speech recognition models we train at Baidu commonly require tens of exaflops of work to train. The long answer is that there are actually many performance limits, including compute, memory bandwidth, communication bandwidth/latency, memory size, and several others. Some neural net models are more sensitive to specific limits than others, and increasing computational throughput alone would benefit some types of models more than others. One of the takeaways from this work is that recurrent neural networks are actually more compute intensive than they may appear to be because they reuse their parameters over many time steps, and the reuse factors are actually higher for RNNs commonly used in speech recognition than the convolutions commonly used in image recognition.