100% agree with this. Although mini-batch SGD is much more parallel than most problems. the scientific codes which parallelize poorly often parallelize poorly

100% agree with this. Although mini-batch SGD is much more parallel than most problems. the scientific codes which parallelize poorly often parallelize poorly because they are written in ancient languages with support for whatever supercomputer interconnect bolted on poorly, Whereas TPU's + JAX have beautiful functional abstractions for distributed tensor computations. Just funding re-writes of all the basic math/physics stack into a language with a PORTABLE parallel functional design and perhaps a compilation layer would definitely get more basic science done than this thing.