3 ms·

you only know the mean is good enough if you examine the spread and/or use a robust estimator. spikes happen in performance (for better and for worse) all the

by dododo 16y ago

you only know the mean is good enough if you examine the spread and/or use a robust estimator. spikes happen in performance (for better and for worse) all the time. one spike is enough to totally distort the mean, no matter how often you repeat. it might be a microbenchmark but to paraphrase: if it's not done well, it's not worth doing. the complete opposite of what the microbenchmark purports to show could be true most of the time.

viraptor 16y ago
I agree with you from the point of view of people doing proper statistics. Mean is not good enough. Then again - let's say I'm a user of some library and I trust the developers to do the right thing and test the new implementation against the old one. The thing that is of interest to me is whether on average on my random test I still see the difference. He did. Sure - the whole test is flawed, may be incorrect, etc. In reality - he just verified something he already knew/expected against his random data. No matter how much we care about correctness, he's still right on some level. There's a visible improvement even on this random test and if he tried it at least 3 times we can assume that the new GC is worth looking at if you had issues with the previous one. "if it's not done well, it's not worth doing"? I'd say - if it answers your question, it's worth doing. If you assume the question was - "can I verify that the new GC is better in my random not-scientific test on a specific dataset", then everything's all right. And I do hope that the person writing the new GC will publish some better stats.
- dododo 16y ago
  1. the point of doing statistics properly is to try to remove your bias about what you think the results are: every developer who has worked hard enough on a problem, probably thinks they made an improvement. it's a good idea to test this, statistically, as you might be surprised. 2. the article didn't repeat anything, at all. it's not even on average (at least, as it's reported). as far as i can tell, he ran a test once and reported the result. 3. the point is that judging from one sample or a mean on a small sample does not answer any reasonable question because the (unrepresented) uncertainty in the answer could actually mean the result goes either way in general. btw, what i'm suggesting isn't some deep, complicated statistics! i'm suggesting the most basic, elementary statistics. this is super, super basic stuff. high school kids here know about the mean and standard deviation here. if you want to be more principled about things kind of thing, you could either do some kind of hypothesis test (e.g., a two-sample t-test adjusting for variance would be appropriate here) or use a Bayes factor to compare the two hypothese (better, not better).

by dododo 16y ago

viraptor 16y ago

I agree with you from the point of view of people doing proper statistics. Mean is not good enough. Then again - let's say I'm a user of some library and I trust the developers to do the right thing and test the new implementation against the old one. The thing that is of interest to me is whether on average on my random test I still see the difference. He did. Sure - the whole test is flawed, may be incorrect, etc. In reality - he just verified something he already knew/expected against his random data. No matter how much we care about correctness, he's still right on some level. There's a visible improvement even on this random test and if he tried it at least 3 times we can assume that the new GC is worth looking at if you had issues with the previous one. "if it's not done well, it's not worth doing"? I'd say - if it answers your question, it's worth doing. If you assume the question was - "can I verify that the new GC is better in my random not-scientific test on a specific dataset", then everything's all right. And I do hope that the person writing the new GC will publish some better stats.

dododo 16y ago
1. the point of doing statistics properly is to try to remove your bias about what you think the results are: every developer who has worked hard enough on a problem, probably thinks they made an improvement. it's a good idea to test this, statistically, as you might be surprised. 2. the article didn't repeat anything, at all. it's not even on average (at least, as it's reported). as far as i can tell, he ran a test once and reported the result. 3. the point is that judging from one sample or a mean on a small sample does not answer any reasonable question because the (unrepresented) uncertainty in the answer could actually mean the result goes either way in general. btw, what i'm suggesting isn't some deep, complicated statistics! i'm suggesting the most basic, elementary statistics. this is super, super basic stuff. high school kids here know about the mean and standard deviation here. if you want to be more principled about things kind of thing, you could either do some kind of hypothesis test (e.g., a two-sample t-test adjusting for variance would be appropriate here) or use a Bayes factor to compare the two hypothese (better, not better).