3 ms·
Measure, but know that measurements may (very frequently) go wrong. In this particular case, there seems no actually measurable difference between two codes. Y
by lifthrasiir 3y ago
Measure, but know that measurements may (very frequently) go wrong.
In this particular case, there seems no actually measurable difference between two codes. You can verify that the actual bytecode change is `c = a2 + c` vs. `c += a2`, but Python int doesn't have a separate implementation for `+=` (`__iadd__`, or `nb_inplace_add` in C) presumably because all integers have to be immutable anyway, so they can't differ that much. [1] And others have pointed out that Python in general doesn't run anything in parallel unless explicitly requested.
Therefore any difference is very likely to be due to the faulty benchmark method. The post suspiciously doesn't mention any benchmarking code (and even what the heck is `square_root_ceil`, which I assumed to be `lambda x: int(math.ceil(math.sqrt(x)))` below), but I did a quick benchmarking and got the following result in a particular environment:
#3, small #3, large #4, small #4, large
----------- ----------- ----------- -----------
min avg min avg min avg min avg
----- ----- ----- ----- ----- ----- ----- -----
0.188 0.228 0.609 0.687 0.203 0.244 0.610 0.695
0.203 0.215 0.610 0.642 0.203 0.211 0.625 0.653
0.203 0.214 0.609 0.641 0.188 0.208 0.609 0.627
0.203 0.213 0.609 0.622 0.187 0.208 0.609 0.630
0.187 0.209 0.609 0.622 0.187 0.208 0.609 0.630
Here I'm calling `factor_fermat3` with small and large arguments, repeat that 10 times (so 20 calls), then do the same for `factor_fermat4`, and then report a single line. I did this 5 times in a row, and it is evident that the first few runs were substantially slower than later.
Modern CPUs vary their frequency to save power, and it takes some (or even much) time for CPUs to adapt to the new workload that demands higher or lower frequency. If I have put `time.sleep` between each run it might be possible that the first run repeats every time, but that would be inaccurate. For multicore systems it's also possible that the core may change over time, with separate power states, so a proper benchmarking needs some core pinning to avoid such issues.
If you only did the first run you may have concluded that #4 was slower than #3, but subsequent runs show that this is not true and they are more or less same especially when you consider the minimum time per call. The average time in comparison is highly variable and you need lots of considerations to stabilize that, which I never did in this case. Also you need statistical testing to conclude that such difference was not a fluke, which is another vast subject. In any case, this post doesn't show any such attempt.
[1] Note that `c += a2` is equivalent to `c = c + a2` and not `c = a2 + c`, which can make a difference when multiprecision arithmetic routines do not account for such cases. I tested both cases and got the same result, however.
- gpderetta 3y agoGood point, measuring is hard. Make sure you are actually know what you are measuring, and if the measurements differ form your expectations, make sure you understand why.
- vlovich123 3y agoYeah, modern CPUs vary frequencies and you have to be very very careful about how you measure to account for this. This had already been true for more than 10 years now and should be muscle memory when benchmarking but it can be easy to overlook and some people don’t know. It doesn’t surprise me that someone hypothesizing that Python automatically executes different lines of code on different cores might make this mistake as they don’t have a good mental model of how the code they run executes on their machine. As you note they probably made other methodological mistakes too.
- jcranmer 3y agoThis really needs to be upvoted harder. The post doesn't say what units the terms a measured in, but it's evident from a quick test that the unit is ms. And when you're in the sub-ms time domain, if you don't know what you're doing, benchmarking is apt to reflect noise in its results.