Benchmarking on the another OS would be much more than benchmarking compilers. Different OS is different OS, it manages memory differently, schedules processes

Benchmarking on the another OS would be much more than benchmarking compilers. Different OS is different OS, it manages memory differently, schedules processes differently. No way it would be a more "clean" benchmark. If you want to use as little code produced by different compiler as possible, you can just link libs you need statically, compiling them by whatever compiler you want (of course it's not impossible that you'll run into obscure compiler-specific errors with that, but whatever, it's doable if you want it that much).