If you're talking about MPI and its use in HPC, you might've been mostly correct 5 years ago. Now, HPC applications at the largest sites are hybrid MPI+OpenMP

If you're talking about MPI and its use in HPC, you might've been mostly correct 5 years ago. Now, HPC applications at the largest sites are hybrid MPI+OpenMP (or some other threading model, or they use GPUs). On Xeon Phi systems like the latest Crays, you pretty much need OpenMP to take advantage of all the cores. The PPC A2 processors on the (formerly #1, now #4) Blue Gene/Q system at LLNL won't get full instruction issue per cycle unless you use multiple hardware threads (the threads are exposed to the hardware and each core will only pull instructions from a single process at a time, so this is somewhat unlike hyperthreading on Intel chips). There can be overheads to running very large numbers of MPI processes, depending on how scalable the MPI implementation is, and there are overheads to having too many threads contend for memory on a single node. At least in the current state of things in HPC, there is a delicate balance to choosing the right process-thread ratio, and it differs from application to application and depends on the node count for the job. People struggle to find the sweet spot, but we're definitely not running all MPI anymore.