Traditionally, supercomputers and mainframes have way higher inter-node IO performance like bandwidth than anything you can spin up on EC2. IO is extremely impo

Traditionally, supercomputers and mainframes have way higher inter-node IO performance like bandwidth than anything you can spin up on EC2. IO is extremely important because it's very rare to have a true shared-nothing algorithm, which means that Amdahl's Law will bite you and make the little bit of IO and coordination a dominant factor. For example, in deep learning, an awful lot of the work is simply moving around and updating data, which has been a bottleneck for attempts to train NNs on multiple GPUs.