3 ms·
Nice to see this. Median is an odd metric to use. I think I might truly be more interested in the mean; maybe the geometric mean. I am also very interested in
by spenczar5 5y ago
Nice to see this.
Median is an odd metric to use. I think I might truly be more interested in the mean; maybe the geometric mean.
I am also very interested in worst-cases with this sort of thing, so the 80th and 95th and 100th percentiles would be helpful.
It would be interesting to see plots of distributions, as well: are boot times really unimodal? They seem like they could easily have multiple modes, which could really mess with all of these measures (but especially the median!).
Because the script is open source (thank you!) I may try this myself.
- cperciva 5y agoI used the median mainly because it was robust against any network glitches which occurred during testing. That said, I did my final measurements from inside EC2 (rather than from my laptop over wifi) so it probably wasn't necessary.
- spenczar5 5y agoNetwork glitches will have a one-sided effect: they will make you slower, but never faster. Trimming out the top 25% slowest measurements, or even just using the minimum, can be more effective. Those are common when profiling programs for exactly that reason - the median execution time for a loop might be influenced by OS weirdness like some background job, but the minimum will hopefully really just measure the loop. Anyway - probably not a big deal, probably mostly academic, although modal behavior would be really interesting.
- cperciva 5y agoYes and no. If there's a network glitch while we're polling to see if the instance is "running", it will result in the "pending-to-running" time being high but the "running-to-port-closed" time being low.