4 ms·
Depends. You can make the same argument about an OS not giving bare-metal performance. For example, disk bandwidth may be different from filesystem performance.
by mirker 5y ago
Depends. You can make the same argument about an OS not giving bare-metal performance. For example, disk bandwidth may be different from filesystem performance. You may have some daemon doing e.g., disk-defragmentation in the background. If you’re trying to be really scientific, you should note any sources of measurement interference, because they always exist. You have to pick your battles.
If you’re running VMs in production, VM performance is the main metric you care about. If not, benchmarking in a container limits performance interference to networking and disk, mostly.
Performance measurements are supposed to be like “real science” experiments. Doing it the “right way” would have controls, multiple trials, and different setups to help control for measurement effects. Unfortunately, most experiments do not have sufficient rigor. The post shows how to document experiment setup, which allows you to replicate, but it may not actually be able to reproduce the hypothesis/conclusion.
- bonzini 5y agoOn top of this, the author is developing KVM itself (more precisely QEMU and Linux virtio drivers); he's not developing with KVM. His work improves performance of the VMs themselves.
- zsmi 5y agoGreat reply. Just to add on: the software environment should be controlled obviously, but the actual environmental conditions around the machine also need to be controlled for an accurate measurement. https://benchmarks.ul.com/support/how-to-benchmark https://benchmarks.ul.com/support/how-to-benchmark "For example, you should test every system in the same location, at room temperature, and away from direct sunlight and other heat sources." And even after all that one should expect a 3% spread in scores.