I think it's both hard to guess things about the code but also the data that's coming in. The "Sufficiently Advanced Compiler" could make a lot of good decision

I think it's both hard to guess things about the code but also the data that's coming in. The "Sufficiently Advanced Compiler" could make a lot of good decisions if it knew ahead of time the shape of the data, e.g. a big uninterrupted batch of fixed transactions. That's why the target for this sort of technology is GPGPU work. The streams of data are very regular and largely a series of batch jobs. For interactive systems with millions of context switches and branches all the "Sufficiently Advanced Compilers" all fall down. There's just not enough commonality between consecutive operations for ahead of time optimizations to occur. Hardware that does great in batch jobs ends up suboptimal for the insanity that is interactive code.