3 ms·
My performance benchmarking workflow
- yjftsjthsd-h 5y agoI'm confused - I thought VMs really needed to be avoided when benchmarking because the host system could completely change how many cycles you're effectively getting and screw up your results without even meaning to. Is there's some good way to control for this that I wasn't aware of? Because benchmarking in virtual machines would be really great for the whole list of reasons if you can get away with it.
- mirker 5y agoDepends. You can make the same argument about an OS not giving bare-metal performance. For example, disk bandwidth may be different from filesystem performance. You may have some daemon doing e.g., disk-defragmentation in the background. If you’re trying to be really scientific, you should note any sources of measurement interference, because they always exist. You have to pick your battles. If you’re running VMs in production, VM performance is the main metric you care about. If not, benchmarking in a container limits performance interference to networking and disk, mostly. Performance measurements are supposed to be like “real science” experiments. Doing it the “right way” would have controls, multiple trials, and different setups to help control for measurement effects. Unfortunately, most experiments do not have sufficient rigor. The post shows how to document experiment setup, which allows you to replicate, but it may not actually be able to reproduce the hypothesis/conclusion.
- bonzini 5y agoOn top of this, the author is developing KVM itself (more precisely QEMU and Linux virtio drivers); he's not developing with KVM. His work improves performance of the VMs themselves.
- zsmi 5y agoGreat reply. Just to add on: the software environment should be controlled obviously, but the actual environmental conditions around the machine also need to be controlled for an accurate measurement. https://benchmarks.ul.com/support/how-to-benchmark https://benchmarks.ul.com/support/how-to-benchmark "For example, you should test every system in the same location, at room temperature, and away from direct sunlight and other heat sources." And even after all that one should expect a 3% spread in scores.
- GrumpyYoungMan 5y agoI avoid VMs when benchmarking for the same reasons you cite. Benchmarking is difficult enough to get reproducible results without adding another variable to an already noisy process.