4 ms·

One of the interesting aspects of benchmarks is that they are usually designed and intended to run in isolation. E.g. if you benchmark a database system, you ex

One of the interesting aspects of benchmarks is that they are usually designed and intended to run in isolation. E.g. if you benchmark a database system, you expect that to be the sole system running on your server machine, in control of all resources. That's not true of software running on desktop systems or mobile phones - desktops usually run many concurrent tasks, so do phones to some degree, and then there's also the question of battery use. That can create skewed incentives if the benchmark isn't carefully designed. E.g. you can usually make space/time tradeoffs regarding performance, so if your benchmark is solely measuring CPU time, it pays off to gobble up all possible RAM for even minor benefits. If your benchmark is only measuring wallclock time, it pays off to gobble up all the CPUs, even if the actual speedup from that is minor. This can lead to software "winning" the benchmark with improvements that are actually detrimental to the performance on end user's systems.

robocat 7y ago
Aside: Chromium includes regression tests for battery usage. I saw a fix get reverted because the code change caused a 5% step increase in battery usage. They have some very good testing infrastructure.
- crispyporkbites 7y ago
  How do they test this?
  - judge2020 7y ago
    this commit[0] might be what's referenced, and this[1] looks to be the test that Google runs (not sure if this is ran in CI, or only done anecdotally for when performance tests[2] show issues). 0: https://github.com/chromium/chromium/commit/208f274e4bcb51748bba83071aa70011ceafa385 https://github.com/chromium/chromium/commit/208f274e4bcb5174... 1: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/refs/heads/master/client/site_tests/power_LoadTest/README.md https://chromium.googlesource.com/chromiumos/third_party/aut... 2: https://chromeperf.appspot.com/ https://chromeperf.appspot.com/
    - saghul 7y ago
      Never heard of [1] before. Impressive stuff.
  - robocat 7y ago
    Here is an example bug that had an unobvious cause (they just reverted the diff out in the end!): https://bugs.chromium.org/p/chromium/issues/detail?id=520952 https://bugs.chromium.org/p/chromium/issues/detail?id=520952 Bisect job status: Completed Bisect job ran on: android_nexus9_perf_bisect ===== BISECT JOB RESULTS ===== Status: Positive: Reproduced a change. Test Command: tools/perf/run_benchmark -v --browser=android-chromium --output-format=buildbot --also-run-disabled-tests power.android_acceptance Test Metric: energy_consumption_mwh/energy_consumption_mwh Relative Change: 150.95% (+/-1.88%) Estimated Confidence: 99.90% Retested CL with revert: No Pretty graphs now deleted.
zakember 7y ago
So unit testing instead of end-user testing at a system level
magicalist 7y ago
Is this responding to something in the post? They mention sampling across different pages and using "in-the-field telemetry", and the sibling post mentions battery testing, so...
- Nitramp 7y ago
  The post mentions that they can drastically reduce memory usage at some CPU time cost in the lite version. V8 is presumably doing the right thing for the end user here, given the in-the-field telemetry etc. It's an instance of carefully weighing different resource consumptions though, where simple benchmarks might drive you to prioritize CPU time at the cost of overall system responsiveness due to memory consumption.

robocat 7y ago

Aside: Chromium includes regression tests for battery usage. I saw a fix get reverted because the code change caused a 5% step increase in battery usage. They have some very good testing infrastructure.

crispyporkbites 7y ago
How do they test this?
- judge2020 7y ago
  this commit[0] might be what's referenced, and this[1] looks to be the test that Google runs (not sure if this is ran in CI, or only done anecdotally for when performance tests[2] show issues). 0: https://github.com/chromium/chromium/commit/208f274e4bcb51748bba83071aa70011ceafa385 https://github.com/chromium/chromium/commit/208f274e4bcb5174... 1: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/refs/heads/master/client/site_tests/power_LoadTest/README.md https://chromium.googlesource.com/chromiumos/third_party/aut... 2: https://chromeperf.appspot.com/ https://chromeperf.appspot.com/
  - saghul 7y ago
    Never heard of [1] before. Impressive stuff.
- robocat 7y ago
  Here is an example bug that had an unobvious cause (they just reverted the diff out in the end!): https://bugs.chromium.org/p/chromium/issues/detail?id=520952 https://bugs.chromium.org/p/chromium/issues/detail?id=520952 Bisect job status: Completed Bisect job ran on: android_nexus9_perf_bisect ===== BISECT JOB RESULTS ===== Status: Positive: Reproduced a change. Test Command: tools/perf/run_benchmark -v --browser=android-chromium --output-format=buildbot --also-run-disabled-tests power.android_acceptance Test Metric: energy_consumption_mwh/energy_consumption_mwh Relative Change: 150.95% (+/-1.88%) Estimated Confidence: 99.90% Retested CL with revert: No Pretty graphs now deleted.

zakember 7y ago

So unit testing instead of end-user testing at a system level

magicalist 7y ago

Is this responding to something in the post? They mention sampling across different pages and using "in-the-field telemetry", and the sibling post mentions battery testing, so...

Nitramp 7y ago
The post mentions that they can drastically reduce memory usage at some CPU time cost in the lite version. V8 is presumably doing the right thing for the end user here, given the in-the-field telemetry etc. It's an instance of carefully weighing different resource consumptions though, where simple benchmarks might drive you to prioritize CPU time at the cost of overall system responsiveness due to memory consumption.