To start, you don't have to do it that way. You can use sampling based profiling so that there is no training set necessary: http://dl.acm.org/citation.cfm?id=2

To start, you don't have to do it that way. You can use sampling based profiling so that there is no training set necessary: http://dl.acm.org/citation.cfm?id=2854044 http://dl.acm.org/citation.cfm?id=2854044 Past that, LLVM is already like this, it's just a matter of shipping bitcode, and outputting the currently optimized bitcode at shutdown. It is actually done in practice, but admittedly, only by those who care tremendously about performance. (IE you don't find it done in common linux distros). Particularly in a datacenter world, where you have hypervisors and things under the covers anyway, it's not a huge deal to have a JIT running. The harder part is actually, if you are a cloud provider, getting anyone to bring bitcode with them :) Even if you promise them X% performance gain to start, where X is usually 10-30%, they don't usually care enough to modify their workflow.