7 ms·

How does an organization which has been built on 99% not doing optimization recognize the one percent where it matters a lot?

eropple 4y ago
In my experience: a profiler, usually. Just because I can throw down a lot of code quickly doesn't mean I don't have the tools to analyze code when I go "hmm, that seems slow".
- incrudible 4y ago
  That works for big hotspots, but not for the tiny papercuts that make every little thing 10x slower.
  - eropple 4y ago
    Sure. That's a tradeoff you consciously make to get the thing out the door. That's what technical debt is. You pay it down later. (Or you go bankrupt and it doesn't matter anymore.)
    - incrudible 4y ago
      The blog post challenges the assumption that the slower version is any way improving productivity. At least for the given example, I agree.
Turskarama 4y ago
Testing, do you not do testing? Use the software. If it seems slow drill down and find the critical path and optimize it.
- yuliyp 4y ago
  Testing is often not enough. Developers might have a few test cases with a few records, or a few other users. Real users might do more with it, and things which perform fine in testing suddenly turn into real performance bottlenecks when you're loading an order list with a thousand entries in it instead of two.
  - Turskarama 4y ago
    When I said testing that includes integration testing, not just unit tests. We do this exact thing with queries that are known to be complex, run them against databases the same size and complexity as real production databases. It's not hard.
    - rerdavies 4y ago
      Building representative databases is not straightforward. If you're lucky enough to eat your own dogfood, run unit tests against a copy(!) of your in-house database. A utility to anonymize the data was a fairly significant investment, but absolutely worth it in the long run. Being able to run and monitor benchmark unit tests for selected critical operations on enterprise-scale test data as part of the continuous-build process: fabulous!
julik 4y ago
By someone presenting plausible evidence that not having that 1% optimised costs the organization money/business, and leadership that would listen
saagarjha 4y ago
In any organization it generally pays off to have most people have a basic knowledge of a subject and then hire domain experts to drive most of the impact. For security, for example, this typically manifests as having very general "best practices" for most developers to follow and then a small team that handles anything that requires advanced understanding of the area. How this typically works with performance is very similar, with a small team working to identify problematic areas where optimization would drive the highest impact, and the rest of the organization keeping performance in mind but not otherwise concerned with it in their day-to-day work.
simplotek 4y ago
> How does an organization which has been built on 99% not doing optimization recognize the one percent where it matters a lot? If you can't recognize the percent where it matters a lot, clearly that percent does not exist and you're bothering about nothing. I see a lot of talk about the importance of tuning Formula 1 cars in a world where everyone drives ford fiestas.
rerdavies 4y ago
Profiling. If you're not profiling, you're completely wasting your time. The 1% is almost never where you think it is. And when you do identify the 1%, you need to be testing optimizations with a profiler constantly while optimizing. Profile. Do some optimization. Profile again. Roll back if not successful. Repeat until done. It's impossible to optimize well if you're not doing profiling. The ultimate tools would be either Intel's profiling tools, or ARM's profiling suite (both very expensive). But MSVC and GCC do a fantastic job of scheduling instructions to avoid pipeline stalls these days, so these deep profiling tools are unlikely to gain more than 2 or 3% performance increases these days. (Worthwhile pretty much only if you're writing GPU drivers for NVidia or AMD). Taken as a given that 100% of your code is at least algorithmically correct in the first place. (Appropriately better than O(N^2) whenever possible). - Former writer of graphics drivers, currently audio DSP engineer.
- anonymoushn 4y ago
  A hash table that accesses a minimum of 2 cache lines per query can be algorithmically correct and also admit a 100% speedup.
- themulticaster 4y ago
  I'm not familiar with how it compares to ARM/Intel's profiling tools, but I found the Linux perf suite to be very capable (though limited to Linux obviously). And Hotspot [1] allows effortless profile visualization using flame graphs, including some very interesting features such as off-CPU time profiling [2]. "perf record" coupled with Hotspot forms a very smooth edit-compile-profile cycle. [1] https://github.com/KDAB/hotspot https://github.com/KDAB/hotspot [2] https://github.com/KDAB/hotspot#off-cpu-profiling https://github.com/KDAB/hotspot#off-cpu-profiling
  - rerdavies 4y ago
    Agreed. Linux perf tools are perfectly acceptable for all but the most ultra-extreme optimization tasks. VTune allows you to determine where pipeline stalls are occurring at the instruction level (for that last 2 or 3% gain in performance). I haven't worked with ARM profilers (way out of my price range), but I assume, given the exorbitant price, they provide the same sort of in-depth analysis. Probably a handful of people on the planet that need that kind of in-depth analysis.

eropple 4y ago

In my experience: a profiler, usually. Just because I can throw down a lot of code quickly doesn't mean I don't have the tools to analyze code when I go "hmm, that seems slow".

incrudible 4y ago
That works for big hotspots, but not for the tiny papercuts that make every little thing 10x slower.
- eropple 4y ago
  Sure. That's a tradeoff you consciously make to get the thing out the door. That's what technical debt is. You pay it down later. (Or you go bankrupt and it doesn't matter anymore.)
  - incrudible 4y ago
    The blog post challenges the assumption that the slower version is any way improving productivity. At least for the given example, I agree.

Turskarama 4y ago

Testing, do you not do testing? Use the software. If it seems slow drill down and find the critical path and optimize it.

yuliyp 4y ago
Testing is often not enough. Developers might have a few test cases with a few records, or a few other users. Real users might do more with it, and things which perform fine in testing suddenly turn into real performance bottlenecks when you're loading an order list with a thousand entries in it instead of two.
- Turskarama 4y ago
  When I said testing that includes integration testing, not just unit tests. We do this exact thing with queries that are known to be complex, run them against databases the same size and complexity as real production databases. It's not hard.
  - rerdavies 4y ago
    Building representative databases is not straightforward. If you're lucky enough to eat your own dogfood, run unit tests against a copy(!) of your in-house database. A utility to anonymize the data was a fairly significant investment, but absolutely worth it in the long run. Being able to run and monitor benchmark unit tests for selected critical operations on enterprise-scale test data as part of the continuous-build process: fabulous!

julik 4y ago

By someone presenting plausible evidence that not having that 1% optimised costs the organization money/business, and leadership that would listen

saagarjha 4y ago

In any organization it generally pays off to have most people have a basic knowledge of a subject and then hire domain experts to drive most of the impact. For security, for example, this typically manifests as having very general "best practices" for most developers to follow and then a small team that handles anything that requires advanced understanding of the area. How this typically works with performance is very similar, with a small team working to identify problematic areas where optimization would drive the highest impact, and the rest of the organization keeping performance in mind but not otherwise concerned with it in their day-to-day work.

simplotek 4y ago

> How does an organization which has been built on 99% not doing optimization recognize the one percent where it matters a lot? If you can't recognize the percent where it matters a lot, clearly that percent does not exist and you're bothering about nothing. I see a lot of talk about the importance of tuning Formula 1 cars in a world where everyone drives ford fiestas.

rerdavies 4y ago

Profiling. If you're not profiling, you're completely wasting your time. The 1% is almost never where you think it is. And when you do identify the 1%, you need to be testing optimizations with a profiler constantly while optimizing. Profile. Do some optimization. Profile again. Roll back if not successful. Repeat until done. It's impossible to optimize well if you're not doing profiling. The ultimate tools would be either Intel's profiling tools, or ARM's profiling suite (both very expensive). But MSVC and GCC do a fantastic job of scheduling instructions to avoid pipeline stalls these days, so these deep profiling tools are unlikely to gain more than 2 or 3% performance increases these days. (Worthwhile pretty much only if you're writing GPU drivers for NVidia or AMD). Taken as a given that 100% of your code is at least algorithmically correct in the first place. (Appropriately better than O(N^2) whenever possible). - Former writer of graphics drivers, currently audio DSP engineer.

anonymoushn 4y ago
A hash table that accesses a minimum of 2 cache lines per query can be algorithmically correct and also admit a 100% speedup.
themulticaster 4y ago
I'm not familiar with how it compares to ARM/Intel's profiling tools, but I found the Linux perf suite to be very capable (though limited to Linux obviously). And Hotspot [1] allows effortless profile visualization using flame graphs, including some very interesting features such as off-CPU time profiling [2]. "perf record" coupled with Hotspot forms a very smooth edit-compile-profile cycle. [1] https://github.com/KDAB/hotspot https://github.com/KDAB/hotspot [2] https://github.com/KDAB/hotspot#off-cpu-profiling https://github.com/KDAB/hotspot#off-cpu-profiling
- rerdavies 4y ago
  Agreed. Linux perf tools are perfectly acceptable for all but the most ultra-extreme optimization tasks. VTune allows you to determine where pipeline stalls are occurring at the instruction level (for that last 2 or 3% gain in performance). I haven't worked with ARM profilers (way out of my price range), but I assume, given the exorbitant price, they provide the same sort of in-depth analysis. Probably a handful of people on the planet that need that kind of in-depth analysis.