3 ms·
> All tests were run on AWS from an m3.medium EC2 instance so you're comparing running times measured on a shared host? This is generally not considered a best
by lorenzhs 12y ago
> All tests were run on AWS from an m3.medium EC2 instance
so you're comparing running times measured on a shared host? This is generally not considered a best practice if you want meaningful numbers.
- p8952 12y agom3.medium is shared in the sense that it is a virtualized instance on top of a hypervisor, however unlike the t2 instance range it provides set resources not burstable ones. Perhaps not the best practice, but it's not an uncommon way of doing benchmarks. TechEmpower Framework Benchmarks[1] are run on AWS for instance. 1: http://www.techempower.com/benchmarks/#section=data-r9&hw=ec2&test=json http://www.techempower.com/benchmarks/#section=data-r9&hw=ec...
- DannyBee 12y agoWhile not uncommon, the performance variation of most of these virtualized instances is so high that your numbers are probably within the range of acceptable error. (For reference, I have a team that does nothing but maintain a performance benchmarking harness and test lab, so i know these things pretty cold :P) I also don't see that the benchmarks were even run multiple times. I tried to follow the ruby-benchmark-suite all the way down the rabbit hole, but don't see anywhere it runs benchmarks multiple times, let alone gives you the variability, etc.