3 ms·

> There's no mystery as to whether you improved something I'd nitpick a little bit and say it's possible that an optimization in one case causes a slowdown in

> There's no mystery as to whether you improved something I'd nitpick a little bit and say it's possible that an optimization in one case causes a slowdown in another case- or worse, a bug. Benchmarks can also be inconsistent on the "same" case due to caches, etc, some of which may live outside of the code you actually control. Even the simplest program will vary a bit when you re-run it on the same system due to the state of that system (other processes, temperature, CPU cache, etc). Some optimizations are clear wins, but many of them involve trade-offs and can have some mystery. Thorough testing/benchmarking helps a lot, but it can only get you so far.

m4rtink 6y ago
Not to mention maintenance and future development costs if the optimization makes the piece of software more complicated and less flexible.
jeffbee 6y ago
FYI Chrome performance is not generally guided by microbenchmarks, for the exact reasons you mention. It is guided by full-scale benchmarks (e.g. render the top 10000 sites) and by ChromeOS-wide profiles gathered in the wild. If a performance change doesn't work in the wild as indicated by profiles then it generally will be backed out and reconsidered. This is consistent with Google's backend performance culture where microbenchmarks are fine and good but changes need to be vetted on a full-scale production loadtest fixture.
- brundolf 6y ago
  Yep, makes sense to me. Unit tests assume that a clean-room environment translates reasonably well to the end result; the more naturally complex or unruly a product or a target metric is, the more your testing process should lean towards integration/real-world testing.
  - throwawayg5TS 6y ago
    It looks like you are confusing testing and benchmarking. Those are different problems that require different skills and tools. Integration/end-to-end tests are terrible for performance benchmarks. They are designed to capture functional issues at the seams. They are usually heavy and not very diverse (because very fragile and/or expensive to maintain), usually focused on a few key critical functional paths, e.g. "can I put something in my shopping cart and pay". That's pretty much the opposite of real-world". The "real-world" is the distribution of what the end-user experiences. With proper tracing, one can identify the key real-world hotspots (including the data associated to it) and then focus on optimizing that.

m4rtink 6y ago

Not to mention maintenance and future development costs if the optimization makes the piece of software more complicated and less flexible.

jeffbee 6y ago

FYI Chrome performance is not generally guided by microbenchmarks, for the exact reasons you mention. It is guided by full-scale benchmarks (e.g. render the top 10000 sites) and by ChromeOS-wide profiles gathered in the wild. If a performance change doesn't work in the wild as indicated by profiles then it generally will be backed out and reconsidered. This is consistent with Google's backend performance culture where microbenchmarks are fine and good but changes need to be vetted on a full-scale production loadtest fixture.

brundolf 6y ago
Yep, makes sense to me. Unit tests assume that a clean-room environment translates reasonably well to the end result; the more naturally complex or unruly a product or a target metric is, the more your testing process should lean towards integration/real-world testing.
- throwawayg5TS 6y ago
  It looks like you are confusing testing and benchmarking. Those are different problems that require different skills and tools. Integration/end-to-end tests are terrible for performance benchmarks. They are designed to capture functional issues at the seams. They are usually heavy and not very diverse (because very fragile and/or expensive to maintain), usually focused on a few key critical functional paths, e.g. "can I put something in my shopping cart and pay". That's pretty much the opposite of real-world". The "real-world" is the distribution of what the end-user experiences. With proper tracing, one can identify the key real-world hotspots (including the data associated to it) and then focus on optimizing that.