5 ms·

And this, ladies and gentlemen, is how you not make speed comparisons. #7 surprised me: why would 'if a:' be slower than 'if not a:'? So I ran the code myself,

And this, ladies and gentlemen, is how you not make speed comparisons. #7 surprised me: why would 'if a:' be slower than 'if not a:'? So I ran the code myself, and ran it a few times. And lo and behold, the third digit varies greatly between runs. And the timing between the first and second variant only differ in the third digit. I ran the first and second variant ten times, and got these values: first: 0.138 +- 0.05 second: 0.137 +- 0.04 So there is no significant difference in my timings, though individual runs differed as much as 0.132 vs. 0.150. Though kudos for providing the full code and the raw output. It made quite easy to reproduce! These are the timings I measured, in octave/matlab syntax: t1 = [0.132781028748, 0.140866041183, 0.13587808609, 0.138177871704, 0.137129068375, 0.150414943695, 0.138676166534, 0.137069940567, 0.13484787941, 0.136043071747]; mean(t1) std(t1) t2 = [ 0.134783983231, 0.143576860428, 0.138277769089, 0.142880916595, 0.140572071075, 0.136868000031, 0.131128787994, 0.134744167328, 0.132730007172, 0.137500047684]; mean(t2) std(t2)

thomasahle 12y ago
I was going to post this: If you do an average of `N` independent trials, your confidence interval is `O(1/sqrt(N))`. Hence with the 10^6 trials done in the article, we must expect a couple of points deviation on the third digit.
- perlgeek 12y ago
  ... and since you don't know how big the constant factor is, you still have to do the experiment a few times. Also there might be factors (that you are not aware of) that differ between several executions of the same program (maybe address space randomization got extremely lucky or unlucky in one execution? Or your OS has a bug the prioritizes processes differently when the PID is a power of 2? Who knows?), or some background processes (cron?) woke up, so even if your number of in-process repetitions is sufficiently high, you still run the thing a few times and observe if the numbers are stable.
  - Matumio 12y ago
    Don't forget CPU frequency, which may slowly step up during the first few runs.
    - stefantalpalaru 12y ago
      You're not supposed to run benchmarks on anything but the 'performance' CPU governor.
      - ambrop7 12y ago
        Though, sometimes you want to benchmark with 'ondemand' just to see how the cpu scaling affects your code. It is known that 'ondemand' may make multi-threaded programs run slower when run on multi-core system. The problem which I have myself observed and reproduced is when two threads are alternately scheduled to run - one thread runs for some time, then wakes up the other and goes to sleep, ad infinitum. With such a workload, the scheduler/governor may enter a situation where the workload moves from one CPU core to the other (*), with no core reaching enough average load to switch it to a higher frequency. I can't find a reference for this, but it was easy to reproduce by writing a simple program with pthreads doing what is described above. My actual application where I encountered this problem consisted of an event loop thread and a worker thread doing computational tasks (encryption). The most interesting observation was how disabling one of the two CPU cores (yes Linux can do that at runtime) sped up my application and benchmark considerably.
- Matumio 12y ago
  Taking the average is a bad idea for performance benchmarks like this running under an operating system. The minimum is much more stable. (Easy to verify experimentally: do 10x10 runs of a short benchmark and compare the variance of 10 minima to the variance of 10 averages.)
  - stefantalpalaru 12y ago
    True, but the minimum might also be dependent on the various caches being primed from previous runs of the same code - a condition that will not appear during normal use.
    - Matumio 12y ago
      Caches being primed is a concern, but not when choosing between minimum and average. Neither is affected much if the first run takes longer than the other nine.
      - gcr 12y ago
        IPython's %timeit magic works by running the function in three loops. It returns the minimum of the three averages. This seems like a good compromise.
rhgraysonii 12y ago
A single read from disk destroys all of them Source: http://www.dis.uniroma1.it/~demetres/didattica/ae/upload/lectures/dean-keynote-ladis2009.pdf http://www.dis.uniroma1.it/~demetres/didattica/ae/upload/lec... , slide 24
Obi_Juan_Kenobi 12y ago
I don't think it's implied that all of the listed differences are significant. I think it's designed to show you that sometimes alternatives make a difference, and other times they simply do not. It makes the page a good resource if you're wondering if doing something a certain way has a performance impact. If it's all the same to use {} vs. dict(), why not be in the habit of using the one that's a little faster? Moreover, the tests are organized in a meaningful way. Tests 13 and 14 are quite interesting when seen right next to each other. Showing just test 13 could be deceptive, leading you to think that operators are faster than methods. Seeing 14 right afterward, however, shows you that isn't the case, and provides some clues as to why each works the way it does. It's just enough of a hint to make you curious about what's actually going on, which I really enjoy. I don't think the fault is with the author, but with your assumptions about the intent. You may argue that exposition would improve this, but I think the lack of explanation is not only elegant, but better at providing insight for the reader. I don't think there's a right answer here, though.
- perlgeek 12y ago
  > I don't think it's implied that all of the listed differences are significant. I think it's designed to show you that sometimes alternatives make a difference, and other times they simply do not. The heading, "Python: faster way" doesn't suggest that to me. And when I see comments like https://news.ycombinator.com/item?id=8814309 https://news.ycombinator.com/item?id=8814309 I feel I'm not the only one who came to conclusion that the first column was meant to be the fastest. And how should the reader judge the significance of timing differences without knowing any variance or spreading of the measurements? Maybe some short headings like "80% faster", "no significant difference" would make things clearer. Or color coding.

thomasahle 12y ago

I was going to post this: If you do an average of `N` independent trials, your confidence interval is `O(1/sqrt(N))`. Hence with the 10^6 trials done in the article, we must expect a couple of points deviation on the third digit.

perlgeek 12y ago
... and since you don't know how big the constant factor is, you still have to do the experiment a few times. Also there might be factors (that you are not aware of) that differ between several executions of the same program (maybe address space randomization got extremely lucky or unlucky in one execution? Or your OS has a bug the prioritizes processes differently when the PID is a power of 2? Who knows?), or some background processes (cron?) woke up, so even if your number of in-process repetitions is sufficiently high, you still run the thing a few times and observe if the numbers are stable.
- Matumio 12y ago
  Don't forget CPU frequency, which may slowly step up during the first few runs.
  - stefantalpalaru 12y ago
    You're not supposed to run benchmarks on anything but the 'performance' CPU governor.
    - ambrop7 12y ago
      Though, sometimes you want to benchmark with 'ondemand' just to see how the cpu scaling affects your code. It is known that 'ondemand' may make multi-threaded programs run slower when run on multi-core system. The problem which I have myself observed and reproduced is when two threads are alternately scheduled to run - one thread runs for some time, then wakes up the other and goes to sleep, ad infinitum. With such a workload, the scheduler/governor may enter a situation where the workload moves from one CPU core to the other (*), with no core reaching enough average load to switch it to a higher frequency. I can't find a reference for this, but it was easy to reproduce by writing a simple program with pthreads doing what is described above. My actual application where I encountered this problem consisted of an event loop thread and a worker thread doing computational tasks (encryption). The most interesting observation was how disabling one of the two CPU cores (yes Linux can do that at runtime) sped up my application and benchmark considerably.
Matumio 12y ago
Taking the average is a bad idea for performance benchmarks like this running under an operating system. The minimum is much more stable. (Easy to verify experimentally: do 10x10 runs of a short benchmark and compare the variance of 10 minima to the variance of 10 averages.)
- stefantalpalaru 12y ago
  True, but the minimum might also be dependent on the various caches being primed from previous runs of the same code - a condition that will not appear during normal use.
  - Matumio 12y ago
    Caches being primed is a concern, but not when choosing between minimum and average. Neither is affected much if the first run takes longer than the other nine.
    - gcr 12y ago
      IPython's %timeit magic works by running the function in three loops. It returns the minimum of the three averages. This seems like a good compromise.

rhgraysonii 12y ago

A single read from disk destroys all of them Source: http://www.dis.uniroma1.it/~demetres/didattica/ae/upload/lectures/dean-keynote-ladis2009.pdf http://www.dis.uniroma1.it/~demetres/didattica/ae/upload/lec... , slide 24

Obi_Juan_Kenobi 12y ago

I don't think it's implied that all of the listed differences are significant. I think it's designed to show you that sometimes alternatives make a difference, and other times they simply do not. It makes the page a good resource if you're wondering if doing something a certain way has a performance impact. If it's all the same to use {} vs. dict(), why not be in the habit of using the one that's a little faster? Moreover, the tests are organized in a meaningful way. Tests 13 and 14 are quite interesting when seen right next to each other. Showing just test 13 could be deceptive, leading you to think that operators are faster than methods. Seeing 14 right afterward, however, shows you that isn't the case, and provides some clues as to why each works the way it does. It's just enough of a hint to make you curious about what's actually going on, which I really enjoy. I don't think the fault is with the author, but with your assumptions about the intent. You may argue that exposition would improve this, but I think the lack of explanation is not only elegant, but better at providing insight for the reader. I don't think there's a right answer here, though.

perlgeek 12y ago
> I don't think it's implied that all of the listed differences are significant. I think it's designed to show you that sometimes alternatives make a difference, and other times they simply do not. The heading, "Python: faster way" doesn't suggest that to me. And when I see comments like https://news.ycombinator.com/item?id=8814309 https://news.ycombinator.com/item?id=8814309 I feel I'm not the only one who came to conclusion that the first column was meant to be the fastest. And how should the reader judge the significance of timing differences without knowing any variance or spreading of the measurements? Maybe some short headings like "80% faster", "no significant difference" would make things clearer. Or color coding.