The point of out-of-order execution is effectively to be able to run parallel computations that don't use the same underlying hardware, eg you can run an intege

The point of out-of-order execution is effectively to be able to run parallel computations that don't use the same underlying hardware, eg you can run an integer operation and a floating-point operation in parallel because they use different functional units, or even two integer operations in parallel if you have redundant integer functional units. Branch prediction / speculative execution then takes that further by making an assumption of which arm of the branch will be taken and executing the instructions in that arm too. When the instruction for the branch completes fully, and the prediction turned out to be correct, the processor continues as normal. If the prediction turned out to be wrong, the processor throws away the pending changes from the incorrectly speculated instructions and starts again from the correct branch arm. An in-order CPU waits for each instruction to finish before it processes the next one. There's no way to work around that in software. At best the hardware can insert latches into the ALU etc so that instructions take the smallest number of clock cycles that they need (eg multiply takes six cycles but add takes only one). The alternative is VLIW ISAs that rely on the compiler to encode multiple instructions that can execute in parallel into a single instruction. That didn't work out well in the past for general computing as demonstrated by Itanium etc, though it's still used for restricted domains like DSPs.