3 ms·

In the past when I have written basic benchmarking code like the first example, the compiler will typically optimize all the code away. I am curious why this d

In the past when I have written basic benchmarking code like the first example, the compiler will typically optimize all the code away. I am curious why this did not happen or what optimization flags were passed in. If it was prevented optimization through flags, is this a fair test?

JoeBot3000 5y ago
For lower numbers this completely optimizes away for me too, but after a certain point the loops stays in place (with no flags)
- rileymat2 5y ago
  Thanks.
jeroenhd 5y ago
I've run the C example code through Godbolt with -O2 and... it removed all the loops because the result wasn't used. Then I added a printf statement and the algorithm came out without too many strangeness. When I reduce the amount of iterations, the compiler inserts a constant where the calculation would've taken place. The amount of iterations in the benchmark seems to be too high for the compiler to evaluate and optimize out. The clang seems to stop any full evaluation at exactly 101 iterations, meaning there's probably a default limit somewhere puts the limit on a nice, round 100. I've taken the code, increased the amount of loops by a factor of 10 (to make the differences more pronounced), added a printf/console.log to make sure the loops themselves don't get thrown away (clang does that with -O2) and on my laptop (i7-10750H) the C code, compiled with clang 12, runs 10000000000 iterations in 9.462s whereas the same number of iterations in Node 16 runs in 19.990s. That's more than a 2x execution duration, with more than a 100% speed difference. For comparison: Java runs in about 9.846s, from the command line (java code.java, no compilation step). C# (dotnet 5) runs in about 9.736s after compilation; compilation takes about a second. Rust runs in about 9.435s after about 0.47s of compilation. Kotlin runs in 9.640s, but it took a few seconds to be compiled into a JAR first. Python 3.9.7 takes forever, but I think that's because its arbitrary length number implementation is trying to make it output the correct result instead of faking it like the other programs are doing. PHP 8 also didn't really stand a chance at over 90 seconds. JS would probably win in a huge code base consisting of mostly dead code (node_modules, anyone?) where compilation would get in the way of quick edit-run-verify loops, but only starting from scratch without a compiler cache set up. From what I can tell, JS certainly isn't _slow_ like PHP, but it isn't _fast_ either. It's somewhere in between the old interpreters of old and compiled/runtime-JIT'ed languages. If anything, this algorithm benchmark would indicate that you're probably better off running C#/Java rather than C/Rust because of the negligible performance difference with the huge benefits of language safety with no effort. This benchmark is far from normal program code, of course, so it doesn't really prove anything. It should be noted that NodeJS outputs the result as "Infinity" whereas C and other languages creates a value that's clearly been bit-wrapped and overflown quite a bit. This can be an advantage (because Infinity + anything = Infinity) or a disadvantage (float math) but that's just how the language works. Edit: looking at the rest of the code, I've also benchmarked the loop vs functional approach (Node 16, same device). // Generate numbers numbers = []; for (let i = -10000000; i < 10000000; i++) numbers.push(i); lowest = numbers.filter(x => x > 0).sort((a,b) =>a-b)[0]; console.log(lowest); ^ this runs in 1.007s numbers = []; for (let i = -10000000; i < 10000000; i++) numbers.push(i); let lowest = Infinity; for (const i of numbers) { if (i > 0 && i < lowest) { lowest = i; } } console.log(lowest); ^ this runs in about 0.629s I wouldn't call a near 40% speed difference "only 2ns". The functional approach is very comfortable to program in, but it comes at a real cost and should definitely be avoided complex in algorithms. Java handles streams very poorly. However, LINQ is quite fast: var numbers = new long[20000000]; for (var i = -10000000L; i < 10000000L; i++) numbers[i + 10000000L] = i; Console.WriteLine("{0}", numbers.Where(x => x > 0).OrderBy(i => i).First() ); ^ this runs in 0.346 seconds. That doesn't make it quite optimal, though: var numbers = new long[20000000]; for (var i = -10000000L; i < 10000000L; i++) numbers[i + 10000000L] = i; long lowest = 9999999999L; foreach (var i in numbers) { if (i > 0 && i < lowest) lowest = i; } Console.WriteLine("{0}", lowest); ^ This runs in 0.151s These examples aren't very conclusive either. Rust's loop version runs in 0.100s and a shitty iterator version that collects the entire iterator and sorts it runs in 0.130ms. Again, the real fight here seems to be between C# and something closer to the metal.
- tored 5y ago
  PHP $numbers = []; for ($i = -10000000; $i < 10000000; $i++) $numbers[] = $i; $lowest = PHP_INT_MAX; foreach ($numbers as $i) { if ($i > 0 && $i < $lowest) { $lowest = $i; } } 1.08 seconds PHP Optimized $lowest = PHP_INT_MAX; foreach (range(-10000000, 10000000) as $i) { if ($i > 0 && $i < $lowest) { $lowest = $i; } } 0.56 seconds Make sure to enable JIT (tracing) Node for loop version for me took 1.25 seconds (I have a very old rig) php --version PHP 8.0.11 (cli) (built: Sep 21 2021 18:25:57) ( NTS Visual C++ 2019 x64 ) Copyright (c) The PHP Group Zend Engine v4.0.11, Copyright (c) Zend Technologies with Zend OPcache v8.0.11, Copyright (c), by Zend Technologies node --version v16.1.0 Note: I measured time within the script and not outside to avoid any bootup time.
  - jeroenhd 5y ago
    All of my measurements included bootup time but PHP took way longer. I guess I don't have JIT enabled, then, because the code is near identical.

JoeBot3000 5y ago

For lower numbers this completely optimizes away for me too, but after a certain point the loops stays in place (with no flags)

rileymat2 5y ago
Thanks.

jeroenhd 5y ago

I've run the C example code through Godbolt with -O2 and... it removed all the loops because the result wasn't used. Then I added a printf statement and the algorithm came out without too many strangeness. When I reduce the amount of iterations, the compiler inserts a constant where the calculation would've taken place. The amount of iterations in the benchmark seems to be too high for the compiler to evaluate and optimize out. The clang seems to stop any full evaluation at exactly 101 iterations, meaning there's probably a default limit somewhere puts the limit on a nice, round 100. I've taken the code, increased the amount of loops by a factor of 10 (to make the differences more pronounced), added a printf/console.log to make sure the loops themselves don't get thrown away (clang does that with -O2) and on my laptop (i7-10750H) the C code, compiled with clang 12, runs 10000000000 iterations in 9.462s whereas the same number of iterations in Node 16 runs in 19.990s. That's more than a 2x execution duration, with more than a 100% speed difference. For comparison: Java runs in about 9.846s, from the command line (java code.java, no compilation step). C# (dotnet 5) runs in about 9.736s after compilation; compilation takes about a second. Rust runs in about 9.435s after about 0.47s of compilation. Kotlin runs in 9.640s, but it took a few seconds to be compiled into a JAR first. Python 3.9.7 takes forever, but I think that's because its arbitrary length number implementation is trying to make it output the correct result instead of faking it like the other programs are doing. PHP 8 also didn't really stand a chance at over 90 seconds. JS would probably win in a huge code base consisting of mostly dead code (node_modules, anyone?) where compilation would get in the way of quick edit-run-verify loops, but only starting from scratch without a compiler cache set up. From what I can tell, JS certainly isn't _slow_ like PHP, but it isn't _fast_ either. It's somewhere in between the old interpreters of old and compiled/runtime-JIT'ed languages. If anything, this algorithm benchmark would indicate that you're probably better off running C#/Java rather than C/Rust because of the negligible performance difference with the huge benefits of language safety with no effort. This benchmark is far from normal program code, of course, so it doesn't really prove anything. It should be noted that NodeJS outputs the result as "Infinity" whereas C and other languages creates a value that's clearly been bit-wrapped and overflown quite a bit. This can be an advantage (because Infinity + anything = Infinity) or a disadvantage (float math) but that's just how the language works. Edit: looking at the rest of the code, I've also benchmarked the loop vs functional approach (Node 16, same device). // Generate numbers numbers = []; for (let i = -10000000; i < 10000000; i++) numbers.push(i); lowest = numbers.filter(x => x > 0).sort((a,b) =>a-b)[0]; console.log(lowest); ^ this runs in 1.007s numbers = []; for (let i = -10000000; i < 10000000; i++) numbers.push(i); let lowest = Infinity; for (const i of numbers) { if (i > 0 && i < lowest) { lowest = i; } } console.log(lowest); ^ this runs in about 0.629s I wouldn't call a near 40% speed difference "only 2ns". The functional approach is very comfortable to program in, but it comes at a real cost and should definitely be avoided complex in algorithms. Java handles streams very poorly. However, LINQ is quite fast: var numbers = new long[20000000]; for (var i = -10000000L; i < 10000000L; i++) numbers[i + 10000000L] = i; Console.WriteLine("{0}", numbers.Where(x => x > 0).OrderBy(i => i).First() ); ^ this runs in 0.346 seconds. That doesn't make it quite optimal, though: var numbers = new long[20000000]; for (var i = -10000000L; i < 10000000L; i++) numbers[i + 10000000L] = i; long lowest = 9999999999L; foreach (var i in numbers) { if (i > 0 && i < lowest) lowest = i; } Console.WriteLine("{0}", lowest); ^ This runs in 0.151s These examples aren't very conclusive either. Rust's loop version runs in 0.100s and a shitty iterator version that collects the entire iterator and sorts it runs in 0.130ms. Again, the real fight here seems to be between C# and something closer to the metal.

tored 5y ago
PHP $numbers = []; for ($i = -10000000; $i < 10000000; $i++) $numbers[] = $i; $lowest = PHP_INT_MAX; foreach ($numbers as $i) { if ($i > 0 && $i < $lowest) { $lowest = $i; } } 1.08 seconds PHP Optimized $lowest = PHP_INT_MAX; foreach (range(-10000000, 10000000) as $i) { if ($i > 0 && $i < $lowest) { $lowest = $i; } } 0.56 seconds Make sure to enable JIT (tracing) Node for loop version for me took 1.25 seconds (I have a very old rig) php --version PHP 8.0.11 (cli) (built: Sep 21 2021 18:25:57) ( NTS Visual C++ 2019 x64 ) Copyright (c) The PHP Group Zend Engine v4.0.11, Copyright (c) Zend Technologies with Zend OPcache v8.0.11, Copyright (c), by Zend Technologies node --version v16.1.0 Note: I measured time within the script and not outside to avoid any bootup time.
- jeroenhd 5y ago
  All of my measurements included bootup time but PHP took way longer. I guess I don't have JIT enabled, then, because the code is near identical.