It takes a really good developer with vast knowledge to do optimizations such as using "inline assembly" for stuff like SIMD. And even though there are enough g

It takes a really good developer with vast knowledge to do optimizations such as using "inline assembly" for stuff like SIMD. And even though there are enough good developers able to do this, the mother of all problems when developing software is managing complexity. Yes, you can take a subroutine and apply local optimizations on it. Building complex software in assembly that on the whole is better optimized than what a compiler can do is next to impossible. Speaking of SIMD and stuff like it, there are already optimizations that LLVM is doing, but such optimizations are hard to apply ahead of time because (1) if you want to distribute those binaries easily, then you need to compile for the common denominator (which is less of an issue with LLVM) and (2) your programming language sucks. It's not the compiler's fault, but rather your own fault that you're using a programming language so confusing that inferring intent from your code is next to impossible. How can the compiler know that you're sorting freaking numbers if you're specifying exactly how bits move around in memory while doing so? If you're speaking about virtual machines though, there are projects out there for .NET or Scala for instance that can recompile/retarget code at runtime to run with SIMD instructions or on your GPUs if you have any. All you need is a virtual machine that runs bytecode and a programming language (slash compiler) that lets you access at runtime the syntax trees of the routines that you want to optimize and that lets you generate new bytecode. So you can easily shove this kind of optimizations in libraries for special-purpose and descriptive DSLs (e.g. LINQ). Of course, it gets tricky and doing stuff like this at runtime has overhead, but it's better than what 99.99% of developers can do, not to mention that good developers first and foremost ship.