> ...every time someone brought be a program for parallelisation, I could refactor it and get a 100 to 10,000 times speed-up on a serial machine... That is a g

> ...every time someone brought be a program for parallelisation, I could refactor it and get a 100 to 10,000 times speed-up on a serial machine... That is a good summary of my experience working in HPC. Many HPC codes had a strong "just throw hardware at it" vibe with relatively little effort put into maximizing performance of individual nodes. There were a couple cases where the optimized code ran faster on my laptop than the supercomputer with the original code. Expensive network interconnects have diminishing returns if you don't do the software design work. Another recurring issue was that the process nodes for exotic silicon was usually a few generations behind the state-of-the-art for commodity silicon, so even if I could get the exotic silicon to perform at its theoretical limits, the commodity silicon was sufficiently superior at the basics that you lost most of the advantages of the exotic architecture in practice with good performance engineering.