3 ms·

It is shocking how often people suggest dubious code changes to "improve performance" with exactly zero benchmarks, or solely artificial benchmarks. Computers a

It is shocking how often people suggest dubious code changes to "improve performance" with exactly zero benchmarks, or solely artificial benchmarks. Computers and compilers are far more complicated than your naive code analysis implies, you _must_ prove any performance changes with robust, real-world benchmarks, especially if your changes make the code more unsafe or harder to understand.

hyperman1 7y ago
When I was younger, I implemented the 8 queens problem on my 386, and micro optimized the hell out of it. Assembler, cycle counting, disable interrupts, etc... Didn't switch to protected mode, but it came close. Then I found a smarter algorithm, and my qbasic proof of concept was magnitudes faster. Ouch! This kid learned something about the nature of optimization that day. Now Im an old fart but even older farts than me stumble into this.
- acdha 7y ago
  That’s a great lesson to have early in your career, too. My favorite example of that was a memory-bound app where a key function had been implemented in both C and x86 assembly. Someone tested and found that the assembler version had been slight performance loss since (IIRC) after the Pentium Pro, and since nobody had wanted to substantially change the assembly code going C-only meant they actually refactored the data access patterns and saw a substantial improvement.
petr_tik 7y ago
Not to excuse people, who provide no metrics or benchmarks for their "performance optimisations", I want to acknowledge that benchmarking is hard. I hope something like NixOS will enable people to write and publish "reproducible" (in quotes because precise numbers are impossible) benchmarks by enabling you to setup your OS, user space and compilation flags exactly like in the benchmark.
- OJFord 7y ago
  Oh that would be really compelling. I only tried NixOS briefly a while ago, but I think I recall you can switch (actually I think literally `switch`) environment like that pretty easily and back again? So your benchmark environment could actually be pretty minimal, or if you were sharing a bug reproduction it could be devoid of unrelated packages/config in your usual environment.

hyperman1 7y ago

When I was younger, I implemented the 8 queens problem on my 386, and micro optimized the hell out of it. Assembler, cycle counting, disable interrupts, etc... Didn't switch to protected mode, but it came close. Then I found a smarter algorithm, and my qbasic proof of concept was magnitudes faster. Ouch! This kid learned something about the nature of optimization that day. Now Im an old fart but even older farts than me stumble into this.

acdha 7y ago
That’s a great lesson to have early in your career, too. My favorite example of that was a memory-bound app where a key function had been implemented in both C and x86 assembly. Someone tested and found that the assembler version had been slight performance loss since (IIRC) after the Pentium Pro, and since nobody had wanted to substantially change the assembly code going C-only meant they actually refactored the data access patterns and saw a substantial improvement.

petr_tik 7y ago

Not to excuse people, who provide no metrics or benchmarks for their "performance optimisations", I want to acknowledge that benchmarking is hard. I hope something like NixOS will enable people to write and publish "reproducible" (in quotes because precise numbers are impossible) benchmarks by enabling you to setup your OS, user space and compilation flags exactly like in the benchmark.

OJFord 7y ago
Oh that would be really compelling. I only tried NixOS briefly a while ago, but I think I recall you can switch (actually I think literally `switch`) environment like that pretty easily and back again? So your benchmark environment could actually be pretty minimal, or if you were sharing a bug reproduction it could be devoid of unrelated packages/config in your usual environment.