3 ms·
Benchmarking on VMs (especially on EC2) is a no-no. It introduces a lot of other factors that can skew results drastically. There are entire companies built aro
by oomkiller 12y ago
Benchmarking on VMs (especially on EC2) is a no-no. It introduces a lot of other factors that can skew results drastically. There are entire companies built around the idea of variable performance in the cloud. If you want to do a decent benchmark, do it on a real piece of hardware.
- VeejayRampay 12y agoAren't most application hosted in the cloud anyway? I understand how VMs are not ideal, but aren't they more "real-life" than "real hardware" nowadays (not a rhetorical question, genuinely curious).
- nitrogen 12y agoThe problem with benchmarking on a VM is that VM effects may totally dominate code effects. It's sort of like using laptop speakers to mix a dubstep album.
- paukiatwee 12y agoThe issue is the EC2 instances are not all "same" performance/stable, and thus each benchmark will have different results. One of the key thing to remember when doing experiment/benchmark is to control variable, now EC2 performance is variable that you cant control.