4 ms·

> Measure. Measure. Measure. The problem is knowing what to measure. There's another saying "When a measure becomes a target, it ceases to be a good measure."

> Measure. Measure. Measure. The problem is knowing what to measure. There's another saying "When a measure becomes a target, it ceases to be a good measure." As an example from memory, there was a game dev company that celebrated they had maxed out the cores on the PS3. That didn't mean anything though, anyone can max out the cores by filing them with bad code. But hey, their "measurement" told them they had maxed out the machine

tialaramex 1mo ago
> The problem is knowing what to measure. This can be a problem, but much less so because so often we're doing "easy mode" where we don't need a proxy. The "it ceases to be a good measure" is because you're measuring a proxy. You wanted to deliver happiness, you measured wealth because it was easier to measure but seemed correlated and now you've got rich miserable people, oops. But software engineers can often measure the actual thing they want to improve directly, not a proxy and so it cannot cease to be a good measure.
- ahartmetz 1mo ago
  With a few caveats, though they do tend to get fixed over time. For example, frame rate. Higher frame rate is better, the end, right? Wellll... Latency and hitching are annoying to measure, so for a long time, they were pretty much ignored. That has improved and hopefully will continue to improve. There is the bufferbloat initiative, gaming magazines take frame time histograms now, input devices and screens commonly have their latency measured. But latency is still under-measured - for GPUs / GPU drivers, for all software, for games and particularly for websites.
  - tialaramex 1mo ago
    Yeah, frame stutter is definitely one of the cases where somebody may end up targeting a proxy and that saying about targets happens as expected. A game which typically hits 50-60fps but has occasional half second stutters may not fix the stutters if your proxy was "time to render 100 frames" and you throw out outliers - since every stutter will be an outlier.
groundzeros2015 1mo ago
That quote mostly applies to human organizations where the connection between outcome and metric is questionable. Or where incentives unexpectedly change behavior. If your goal is fast software it can be measured quantively and you’re likely to improve the actual thing using those metrics.
- socalgal2 1mo ago
  The goal above was to write fast software. The developers made the mistake of seeing that they had saturated all the cores as "We did a good job of using the entire machine" = "our code is fast". They had a measurement and misinterpreted what it meant.
  - groundzeros2015 1mo ago
    This is just dumb. You wanted to improve performance but measured memory instead? Whoops. Why not measure performance for which there are many good metrics.
    - socalgal2 1mo ago
      They didn't measure memory, they measured how busy each processing unit was. It's like going into Activity Monitor, Tops, Task Manager, seeing all CPUs at 100% from your app and thinking "We've reach peak perf by utilizing all the processing power" People still do that today. They run some LLM or gen AI type app. See all the numbers reach high percentages, and wrongly think "this app is using all of my machine's potential"
      - groundzeros2015 1mo ago
        Yes it was an analogy about measuring two unrelated things.

tialaramex 1mo ago

> The problem is knowing what to measure. This can be a problem, but much less so because so often we're doing "easy mode" where we don't need a proxy. The "it ceases to be a good measure" is because you're measuring a proxy. You wanted to deliver happiness, you measured wealth because it was easier to measure but seemed correlated and now you've got rich miserable people, oops. But software engineers can often measure the actual thing they want to improve directly, not a proxy and so it cannot cease to be a good measure.

ahartmetz 1mo ago
With a few caveats, though they do tend to get fixed over time. For example, frame rate. Higher frame rate is better, the end, right? Wellll... Latency and hitching are annoying to measure, so for a long time, they were pretty much ignored. That has improved and hopefully will continue to improve. There is the bufferbloat initiative, gaming magazines take frame time histograms now, input devices and screens commonly have their latency measured. But latency is still under-measured - for GPUs / GPU drivers, for all software, for games and particularly for websites.
- tialaramex 1mo ago
  Yeah, frame stutter is definitely one of the cases where somebody may end up targeting a proxy and that saying about targets happens as expected. A game which typically hits 50-60fps but has occasional half second stutters may not fix the stutters if your proxy was "time to render 100 frames" and you throw out outliers - since every stutter will be an outlier.

groundzeros2015 1mo ago

That quote mostly applies to human organizations where the connection between outcome and metric is questionable. Or where incentives unexpectedly change behavior. If your goal is fast software it can be measured quantively and you’re likely to improve the actual thing using those metrics.

socalgal2 1mo ago
The goal above was to write fast software. The developers made the mistake of seeing that they had saturated all the cores as "We did a good job of using the entire machine" = "our code is fast". They had a measurement and misinterpreted what it meant.
- groundzeros2015 1mo ago
  This is just dumb. You wanted to improve performance but measured memory instead? Whoops. Why not measure performance for which there are many good metrics.
  - socalgal2 1mo ago
    They didn't measure memory, they measured how busy each processing unit was. It's like going into Activity Monitor, Tops, Task Manager, seeing all CPUs at 100% from your app and thinking "We've reach peak perf by utilizing all the processing power" People still do that today. They run some LLM or gen AI type app. See all the numbers reach high percentages, and wrongly think "this app is using all of my machine's potential"
    - groundzeros2015 1mo ago
      Yes it was an analogy about measuring two unrelated things.