3 ms·

Sales are absolutely gameable and regularly are. Sales on false promises to customers, overpromising feature delivery, overstating financial benefits to the cus

by rfrey 3y ago

Sales are absolutely gameable and regularly are. Sales on false promises to customers, overpromising feature delivery, overstating financial benefits to the customer, are so common as to be cliche. The customers end up being short term relationships, the sales provide negative value to the company due to reputation loss (not realized for perhaps years), but the salesperson collects the commission and is probably working somewhere else when the smoke clears. And I'm not sure if you're claiming "number of users" is ungameable. If so, I'd invite you to reconsider that - we've just lived through an entire era where companies gamed user metrics to harvest sweet, sweet VC dollars. Founders game the metrics for VCs, vice presidents game the metrics for founders, product managers game the metrics for VPs.

osigurdson 3y ago
If cash is gameable, then play the game. If I hire a machine to generate cash and they are consistently able to do and grow that it seems that this is a good game to play. You could also reformulate the objective function to NPV.
- rfrey 3y ago
  The point is that not all cash is the same, and some immediate cash causes medium or long term harm. But if cash is your metric, it will be gamed by not distinguishing between healthy and unhealthy cash. So you modify your metric to try to eliminate the harmful kind. But now you're in the same race as other metrics.
  - osigurdson 3y ago
    If NPV isn’t sufficient then you are essentially in the realm of maximizing happiness as the objective function.
    - rfrey 3y ago
      You're begging the question by assuming we can tell what sales are positive and which are negative value. In other words you're asserting we have perfect knowledge and can't be gamed, to argue the measurement can't be gamed.
      - osigurdson 3y ago
        Here is a thought, introduce a new term "metric NPV correlation factor". If a particular metric has a perfect correlation with NPV it is assigned 1.0. If increasing or decreasing the metric has no impact it is assigned 0.0. Now that we have established a gradient, it is possible to further the discussion. You are stating that cash / sales do not have exactly 1.0 but can be something less. I can accept that. However, the correlation is very high and higher correlation factors are harder to game. Metrics with high correlation factors also do not need to be hidden. Let's contrast this with another metric: number of emails sent per month. Low correlation factor, easy to game, likely to actually cause harm to a business. This is the kind of metric that you would certainly need to invoke Goodhart's law on (i.e. hide) which is, in turn, a sign that a metric should not be used. My main thesis is metrics must have high NPV correlation factors or should not used at all. High levels of scrutiny should be placed on those introducing such metrics. Finally, no metric should ever be hidden as it is a clear sign that NPV correlation is low by definition.

by rfrey 3y ago

osigurdson 3y ago

If cash is gameable, then play the game. If I hire a machine to generate cash and they are consistently able to do and grow that it seems that this is a good game to play. You could also reformulate the objective function to NPV.

rfrey 3y ago
The point is that not all cash is the same, and some immediate cash causes medium or long term harm. But if cash is your metric, it will be gamed by not distinguishing between healthy and unhealthy cash. So you modify your metric to try to eliminate the harmful kind. But now you're in the same race as other metrics.
- osigurdson 3y ago
  If NPV isn’t sufficient then you are essentially in the realm of maximizing happiness as the objective function.
  - rfrey 3y ago
    You're begging the question by assuming we can tell what sales are positive and which are negative value. In other words you're asserting we have perfect knowledge and can't be gamed, to argue the measurement can't be gamed.
    - osigurdson 3y ago
      Here is a thought, introduce a new term "metric NPV correlation factor". If a particular metric has a perfect correlation with NPV it is assigned 1.0. If increasing or decreasing the metric has no impact it is assigned 0.0. Now that we have established a gradient, it is possible to further the discussion. You are stating that cash / sales do not have exactly 1.0 but can be something less. I can accept that. However, the correlation is very high and higher correlation factors are harder to game. Metrics with high correlation factors also do not need to be hidden. Let's contrast this with another metric: number of emails sent per month. Low correlation factor, easy to game, likely to actually cause harm to a business. This is the kind of metric that you would certainly need to invoke Goodhart's law on (i.e. hide) which is, in turn, a sign that a metric should not be used. My main thesis is metrics must have high NPV correlation factors or should not used at all. High levels of scrutiny should be placed on those introducing such metrics. Finally, no metric should ever be hidden as it is a clear sign that NPV correlation is low by definition.