To clarify: "That's 100 cycles during which your CPU does nothing but wait..." is true only in the absolute worst case. In the likely case, your superscalar, ou

To clarify: "That's 100 cycles during which your CPU does nothing but wait..." is true only in the absolute worst case. In the likely case, your superscalar, out-of-order processor is working on the next several instructions, possibly including some speculative instructions after branches, and simultaneous-multithreading is letting another thread use the now-idle resources. Furthermore, since it's superscalar there are tons of compute resources (integer ALUs, FP ALUs, vector ALUs, to mention a few) which the CPU scheduler will try to keep busy as close to 100% of the time as possible. That said, in any performance critical application, you pay close attention to your memory hierarchy and you design and compile your application in a way to respect that as well as take advantage of the underlying hardware architecture and resources to the fullest.