4 ms·

Goodharts law basically states that false proxies are game-able. The solution is to stop wasting time on tracking false proxies. Instead, roll up your sleeves a

Goodharts law basically states that false proxies are game-able. The solution is to stop wasting time on tracking false proxies. Instead, roll up your sleeves and do something.

hinkley 2y ago
Or use the metrics to spot check your processes, and move on to other concerns before the histamine reaction starts. In theory you can return to the metrics later for shorter intervals.
- osigurdson 2y ago
  What are some example metrics that you might track that add value?
  - hinkley 2y ago
    With any new habit in life, you fixate on it for a very brief period and then try to let it fade into the background. Then you periodically revisit it to see how you’re doing. From a programming standpoint, and off the top of my head, I would include TDD, code coverage, and anything that comes out of a root cause analysis. I tell junior devs who ask to spend a little more time on every task than they think necessary, trying to raise their game. When doing a simple task you should practice all of your best intentions and new intentions to build the muscle memory.
    - osigurdson 2y ago
      >> TDD, code coverage I don't know how to track TDD, but for me, code coverage is an example of the same old false proxies that people used to track in the 000s. Before creating a metric and policing it, make sure you can rigorously defend its relationship to NPV. If you can't do this, find something else to track.
- alok-g 2y ago
  Why not define additional metrics around what you would spot check and then produce a consolidated figure of merit as the metric to optimize. Aren't we doing that with AI training and general optimization (including regularization).
stoperaticless 2y ago
> Goodharts law basically states that false proxies are game-able Exactly (btw. very nice way to put it) > stop wasting time on tracking false proxies Some times a proxy is much cheaper. (Medical anlogy of limited depth: Instead of doing a surgery to see stuff in person, one might opt to check some ratios in the blood first)
- osigurdson 2y ago
  >> check some ratios in the blood first This would not count as a false proxy however. The problem in software is, it is very hard to construct meaningful proxy metrics. Most of the time it ends up being tangential to value.
  - stoperaticless 2y ago
    Tangential value is still value :) I agree in principle, just want to add a bit of nuance. Lets take famous “lines of code” metric. It would be counterproductive to reward it (as proxy of productivity). But it is a good metric to know. For the same reason why it’s good to know the weight of ships you produce.
    - osigurdson 2y ago
      "Tangential to value", not "tangential value". The value in tracking false proxies like lines of code, accrues to the tracker, not the customer, business or anyone else. The tracker is able to extract value from the business, essentially by tricking it (and themselves) into believing that such metrics are valuable. It isn't a good use of time in my opinion, but probably a low stress / chill occupation if that is your objective.
godelski 2y ago
> that **false** proxies are game-able. You say this like there are measures that aren't proxies. Tbh I can't think of a single one. Even trivial. All measures are proxies and all measures are gameable. If you are uncertain, host a prize and you'll see how creative people get.
- osigurdson 2y ago
  Try gaming NPV.
  - godelski 2y ago
    This is a joke, right? https://en.wikipedia.org/wiki/Net_present_value#Disadvantages https://en.wikipedia.org/wiki/Net_present_value#Disadvantage...
    - osigurdson 2y ago
      Not at all. Don't conflate NPV predictions vs current NPV where all cashflows are known.
      - godelski 2y ago
        Well for one, it's hard not to conflate when you're expecting me to know what you mean as opposed to what you say. Second off, you don't think it's possible to hide costs, exaggerate revenue, ignore risks, and/or optimize short term revenue in favor of long term? If you think something isn't hackable you just aren't looking hard enough
        osigurdson 2y ago
        What, in your opinion, is any for profit business ultimately trying to (legally) maximize? I'd say it is NPV. That can mean taking risks, optimizing short term revenue in favor of long term (and vice versa).

Goodharts law basically states that false proxies are game-able. The solution is to stop wasting time on tracking false proxies. Instead, roll up your sleeves and do something.

hinkley 2y ago

Or use the metrics to spot check your processes, and move on to other concerns before the histamine reaction starts. In theory you can return to the metrics later for shorter intervals.

osigurdson 2y ago
What are some example metrics that you might track that add value?
- hinkley 2y ago
  With any new habit in life, you fixate on it for a very brief period and then try to let it fade into the background. Then you periodically revisit it to see how you’re doing. From a programming standpoint, and off the top of my head, I would include TDD, code coverage, and anything that comes out of a root cause analysis. I tell junior devs who ask to spend a little more time on every task than they think necessary, trying to raise their game. When doing a simple task you should practice all of your best intentions and new intentions to build the muscle memory.
  - osigurdson 2y ago
    >> TDD, code coverage I don't know how to track TDD, but for me, code coverage is an example of the same old false proxies that people used to track in the 000s. Before creating a metric and policing it, make sure you can rigorously defend its relationship to NPV. If you can't do this, find something else to track.
alok-g 2y ago
Why not define additional metrics around what you would spot check and then produce a consolidated figure of merit as the metric to optimize. Aren't we doing that with AI training and general optimization (including regularization).

stoperaticless 2y ago

> Goodharts law basically states that false proxies are game-able Exactly (btw. very nice way to put it) > stop wasting time on tracking false proxies Some times a proxy is much cheaper. (Medical anlogy of limited depth: Instead of doing a surgery to see stuff in person, one might opt to check some ratios in the blood first)

osigurdson 2y ago
>> check some ratios in the blood first This would not count as a false proxy however. The problem in software is, it is very hard to construct meaningful proxy metrics. Most of the time it ends up being tangential to value.
- stoperaticless 2y ago
  Tangential value is still value :) I agree in principle, just want to add a bit of nuance. Lets take famous “lines of code” metric. It would be counterproductive to reward it (as proxy of productivity). But it is a good metric to know. For the same reason why it’s good to know the weight of ships you produce.
  - osigurdson 2y ago
    "Tangential to value", not "tangential value". The value in tracking false proxies like lines of code, accrues to the tracker, not the customer, business or anyone else. The tracker is able to extract value from the business, essentially by tricking it (and themselves) into believing that such metrics are valuable. It isn't a good use of time in my opinion, but probably a low stress / chill occupation if that is your objective.

godelski 2y ago

> that **false** proxies are game-able. You say this like there are measures that aren't proxies. Tbh I can't think of a single one. Even trivial. All measures are proxies and all measures are gameable. If you are uncertain, host a prize and you'll see how creative people get.

osigurdson 2y ago
Try gaming NPV.
- godelski 2y ago
  This is a joke, right? https://en.wikipedia.org/wiki/Net_present_value#Disadvantages https://en.wikipedia.org/wiki/Net_present_value#Disadvantage...
  - osigurdson 2y ago
    Not at all. Don't conflate NPV predictions vs current NPV where all cashflows are known.
    - godelski 2y ago
      Well for one, it's hard not to conflate when you're expecting me to know what you mean as opposed to what you say. Second off, you don't think it's possible to hide costs, exaggerate revenue, ignore risks, and/or optimize short term revenue in favor of long term? If you think something isn't hackable you just aren't looking hard enough
      - osigurdson 2y ago
        What, in your opinion, is any for profit business ultimately trying to (legally) maximize? I'd say it is NPV. That can mean taking risks, optimizing short term revenue in favor of long term (and vice versa).