What have happened the last 20 years is that the total amount of arithmetic that can be done has increased tremendously, while communication has not to the same

What have happened the last 20 years is that the total amount of arithmetic that can be done has increased tremendously, while communication has not to the same extent. That means that more problems than before is communication bound (even inside a processor) than before. Since problems range from extremly IO bound to entirely CPU-bound, more problems are bound by communication than before. A friend of mine that worked with supercomputers back in the 90s used to say that a supercomputer was a device that converted a computational problem to a communication problem. Today this is true for mainstream computing as well. There are still problems that can take advantage of the massive amount of computing available (Some mentioned Convolutional Neural Networks), but many tasks that used to be CPU bound is now effectively IO bound.