The parallelism is there in most of the algorithms, and it gets taken out by a human translating the algorithm (which is inherently parallel) into some arbitrar

The parallelism is there in most of the algorithms, and it gets taken out by a human translating the algorithm (which is inherently parallel) into some arbitrary language that the computer understands (most of which aren't parallel). If you do a REALLY good job of pattern recognition coupled with some execution in a simulator you might be able to figure out what the core of a program is and go to work on that. I'm skeptical of anything that reverses arbitrary machine code from serial to parallel but that's not to say it can't happen. I did some work patching binaries on "basic blocks" which are chunks of code between branches. Inside a basic block you can reorder instructions however you'd like so long as you don't step on your own feet. The key piece of insight that enables this is that as long as the program state is correct by the time you get to a branch, it doesn't matter the EXACT order in which it got to the correct state. If they've had some kind of bigger insight, it's not unreasonable to come up with short-timescale virtual threads. Cache latencies are on the order of a few to say a hundred cycles. Core to core latency should be about the same if you design it well. Having an insight that's bigger than basic blocks and scales up to say maybe 1000 cycles would enable you to shuffle computation from one core to another and back in time to make an improvement. CPUs which do out-of-order execution already have a commit engine which either commits or throws away the results of speculative computation as a result of the branch predictor. To make something which operates at a larger (though not program-wide) scale isn't impossible to believe.