I've thought about this quite a lot because people often ask me. My main comment is that for one algorithm to be judged against another, the more different env

I've thought about this quite a lot because people often ask me. My main comment is that for one algorithm to be judged against another, the more different environments it is tested in the better, and we can get a deeper understanding of its performance profile. If it's always better then that's clear. If it is sometimes better and sometimes worse, either it's not really better, or there's some dependency on the hardware or other environment: but that is an interesting result in itself. However there is always a place for very hardware specific claims. E.g. "For this chip/motherboard combination this flag is better", and for that we might always have a problem. Interesting thought about the either VM or booting. Somebody suggested to me making live images (as in live dvds) which would serve this purpose.