3 ms·

I personally find performance optimization on an existing, high traffic system to be some of my favorite times as a developer. You have a large number, and you

I personally find performance optimization on an existing, high traffic system to be some of my favorite times as a developer. You have a large number, and you have to get it as close to zero as possible. There's no mystery as to whether you improved something and it has a tangible reward.

segmondy 5y ago
It doesn't even have to be for a high traffic system, it could also be for a cost constrained system. If you're an indie developer. You might be able to afford $50 a month but not $500 a month for your side project and improving performance can keep you in business and give you a shot at success.
- LeonB 5y ago
  I saw a similar but different variation involving a low-use system. Imagine it was a system for detectives to track complex murder* investigations. They only need to conduct approx 10 such investigations per year, but quality/correctness is crucial. The full end to end tests are performed thousands and thousands of time per year, and if they are slow the entire development effort is slow. So you end up needing to make the whole system performant - even though end users aren’t experiencing the pain, devs/QA are. (* this is not the actual domain, but similar rarity + criticality)
brundolf 5y ago
> There's no mystery as to whether you improved something I'd nitpick a little bit and say it's possible that an optimization in one case causes a slowdown in another case- or worse, a bug. Benchmarks can also be inconsistent on the "same" case due to caches, etc, some of which may live outside of the code you actually control. Even the simplest program will vary a bit when you re-run it on the same system due to the state of that system (other processes, temperature, CPU cache, etc). Some optimizations are clear wins, but many of them involve trade-offs and can have some mystery. Thorough testing/benchmarking helps a lot, but it can only get you so far.
- m4rtink 5y ago
  Not to mention maintenance and future development costs if the optimization makes the piece of software more complicated and less flexible.
- jeffbee 5y ago
  FYI Chrome performance is not generally guided by microbenchmarks, for the exact reasons you mention. It is guided by full-scale benchmarks (e.g. render the top 10000 sites) and by ChromeOS-wide profiles gathered in the wild. If a performance change doesn't work in the wild as indicated by profiles then it generally will be backed out and reconsidered. This is consistent with Google's backend performance culture where microbenchmarks are fine and good but changes need to be vetted on a full-scale production loadtest fixture.
  - brundolf 5y ago
    Yep, makes sense to me. Unit tests assume that a clean-room environment translates reasonably well to the end result; the more naturally complex or unruly a product or a target metric is, the more your testing process should lean towards integration/real-world testing.
    - throwawayg5TS 5y ago
      It looks like you are confusing testing and benchmarking. Those are different problems that require different skills and tools. Integration/end-to-end tests are terrible for performance benchmarks. They are designed to capture functional issues at the seams. They are usually heavy and not very diverse (because very fragile and/or expensive to maintain), usually focused on a few key critical functional paths, e.g. "can I put something in my shopping cart and pay". That's pretty much the opposite of real-world". The "real-world" is the distribution of what the end-user experiences. With proper tracing, one can identify the key real-world hotspots (including the data associated to it) and then focus on optimizing that.
GordonS 5y ago
I'm with you - one of my favourite things to do is optimise performance, whether it's memory, CPU, latency, whatever. Actually, I often enjoy it a bit too much... it's frequently the case that I'll realise I've just spent an entire day reducing memory allocations that didn't really need reducing, rather than building features :(
dan-robertson 5y ago
If you work at a sufficiently large company then optimisation work on sufficiently low level systems can save 7–8 figures per year, not that you’d likely see much of those savings yourself. It often turns out that some tiny bit of code with not-great performance becomes embedded deeply in the stack and the small cost can add up over many machines.

segmondy 5y ago

It doesn't even have to be for a high traffic system, it could also be for a cost constrained system. If you're an indie developer. You might be able to afford $50 a month but not $500 a month for your side project and improving performance can keep you in business and give you a shot at success.

LeonB 5y ago
I saw a similar but different variation involving a low-use system. Imagine it was a system for detectives to track complex murder* investigations. They only need to conduct approx 10 such investigations per year, but quality/correctness is crucial. The full end to end tests are performed thousands and thousands of time per year, and if they are slow the entire development effort is slow. So you end up needing to make the whole system performant - even though end users aren’t experiencing the pain, devs/QA are. (* this is not the actual domain, but similar rarity + criticality)

brundolf 5y ago

> There's no mystery as to whether you improved something I'd nitpick a little bit and say it's possible that an optimization in one case causes a slowdown in another case- or worse, a bug. Benchmarks can also be inconsistent on the "same" case due to caches, etc, some of which may live outside of the code you actually control. Even the simplest program will vary a bit when you re-run it on the same system due to the state of that system (other processes, temperature, CPU cache, etc). Some optimizations are clear wins, but many of them involve trade-offs and can have some mystery. Thorough testing/benchmarking helps a lot, but it can only get you so far.

m4rtink 5y ago
Not to mention maintenance and future development costs if the optimization makes the piece of software more complicated and less flexible.
jeffbee 5y ago
FYI Chrome performance is not generally guided by microbenchmarks, for the exact reasons you mention. It is guided by full-scale benchmarks (e.g. render the top 10000 sites) and by ChromeOS-wide profiles gathered in the wild. If a performance change doesn't work in the wild as indicated by profiles then it generally will be backed out and reconsidered. This is consistent with Google's backend performance culture where microbenchmarks are fine and good but changes need to be vetted on a full-scale production loadtest fixture.
- brundolf 5y ago
  Yep, makes sense to me. Unit tests assume that a clean-room environment translates reasonably well to the end result; the more naturally complex or unruly a product or a target metric is, the more your testing process should lean towards integration/real-world testing.
  - throwawayg5TS 5y ago
    It looks like you are confusing testing and benchmarking. Those are different problems that require different skills and tools. Integration/end-to-end tests are terrible for performance benchmarks. They are designed to capture functional issues at the seams. They are usually heavy and not very diverse (because very fragile and/or expensive to maintain), usually focused on a few key critical functional paths, e.g. "can I put something in my shopping cart and pay". That's pretty much the opposite of real-world". The "real-world" is the distribution of what the end-user experiences. With proper tracing, one can identify the key real-world hotspots (including the data associated to it) and then focus on optimizing that.

GordonS 5y ago

I'm with you - one of my favourite things to do is optimise performance, whether it's memory, CPU, latency, whatever. Actually, I often enjoy it a bit too much... it's frequently the case that I'll realise I've just spent an entire day reducing memory allocations that didn't really need reducing, rather than building features :(

dan-robertson 5y ago

If you work at a sufficiently large company then optimisation work on sufficiently low level systems can save 7–8 figures per year, not that you’d likely see much of those savings yourself. It often turns out that some tiny bit of code with not-great performance becomes embedded deeply in the stack and the small cost can add up over many machines.