4 ms·

"You get what you measure" is the bane of KPIs everywhere. Measure deals closed, and you get deals with no regard for profitability. Measure profitability, and

by nikanj 8y ago

"You get what you measure" is the bane of KPIs everywhere. Measure deals closed, and you get deals with no regard for profitability. Measure profitability, and you will burn through your customer base by jacking prices and cutting quality. Etc etc.

tomjen3 8y ago
That is only true if you do that with no regard for the way it was achieved.
- supercheetah 8y ago
  That's kind of the point the point though. A lot of people will lose focus on the reason for a metric.
- acdha 8y ago
  That’s technically correct but it’s like saying we shouldn’t have an obesity crisis because anyone who eats less than they burn will reliably lose weight. We have too much evidence about human nature to treat something which happens so commonly as an edge case.
- setr 8y ago
  the problem with figuring out how the numbers are being achieved is that, without tracking everything totally and fully, you'll be reliant on some kind of estimate/measure. Which can also then be gamed.
personjerry 8y ago
So then you add “size of customer base” to your metrics!
- scj 8y ago
  Doing so in a large software company would likely result in Sales & Marketing being empowered. Which in turn will likely lead to technical debt (as the company stretches to meet impractical promises), as well as stagnation in Engineering (S&M are delivering on the metrics, so they get promoted). In general, large organizations are like an AI. In this case, making a goal that won't backfire when applied without thought is really difficult.
  - nostrademons 8y ago
    That's why the goal in so many large public companies becomes "Make the stock price go up." If the stock price goes up, the executives and everyone else in the company with stock can unload it, take the cash, and let the new shareholders deal with any long-term consequences of their actions. This works great up until it doesn't.
minimaxir 8y ago
When I interviewed for a data analyst position at a known startup, I was asked to define a target KPI for a controlled pseudo-A/B experiment, and I suggested total revenue. The interviewer replied "that's a dumb answer because revenue can be gamed; you should have said # of sales." I was too taken aback to follow up asking why # of sales can't be gamed too.
- bsg75 8y ago
  Number of sales (or customers) is a vanity metric favored by startups looking to entice investment. Revenue is a metric useful for measuring actual business health. Any chance the interviewer was overly exit eager ?
  - minimaxir 8y ago
    It is a startup very well known for being a loss-leader.
- rhizome 8y ago
  Credit where due, I think this is the first time I've read of anybody being told how they got something wrong in an interview.
  - geofft 8y ago
    Why is that credit? Good interview questions don't have a right or wrong answer, they have a right or wrong process of coming up with an answer. Especially for a question like "Pick a metric," almost any relevant metric is defensible and has tradeoffs, and the question should be about how the interviewee analyzes engineering tradeoffs, gathers requirements, defends their initial proposal, and is open to counterarguments. (If there's a correct answer, you can just give them a quiz. If there's a Googleable correct answer, you can just employ them and give them Google.)
- abiox 8y ago
  > interviewer replied "that's a dumb answer a bit ot, but was that the actual response? personally, stuff like this would have me ending the interview early.
heavenlyblue 8y ago
That’s why organisations need leaders who can change the direction having realised that the current state had degraded to being gamed. Seek revenue, then seek new clients but not revenue, then go on R&D spree. People are not going to be happy, but the organisation is going to be healthy. IMO there must be high-level metrics which define which low-level metrics are valuable today.
- usrusr 8y ago
  It's getting weirder when you factor in shareholder value: sacrificing the future of the company for looking good in all the fashionable metrics might be exactly the leaders job, depending on how the board is made up.
mwexler 8y ago
But of course, "If you can't measure it, you can't manage it", said by Drucker, says EVERYONE. Hence the cherry picking of preferential KPIs or OKRs. But, actually, Drucker didn't say it. In fact, W. Edwards Deming said it. And what he actually said was "It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.". So, basically, the quote held up as the "reason" behind excessive over-tracking of the wrong thing... is wrong. https://blog.deming.org/2015/08/myth-if-you-cant-measure-it-you-cant-manage-it/ https://blog.deming.org/2015/08/myth-if-you-cant-measure-it-...

by nikanj 8y ago

tomjen3 8y ago

That is only true if you do that with no regard for the way it was achieved.

supercheetah 8y ago
That's kind of the point the point though. A lot of people will lose focus on the reason for a metric.
acdha 8y ago
That’s technically correct but it’s like saying we shouldn’t have an obesity crisis because anyone who eats less than they burn will reliably lose weight. We have too much evidence about human nature to treat something which happens so commonly as an edge case.
setr 8y ago
the problem with figuring out how the numbers are being achieved is that, without tracking everything totally and fully, you'll be reliant on some kind of estimate/measure. Which can also then be gamed.

personjerry 8y ago

So then you add “size of customer base” to your metrics!

scj 8y ago
Doing so in a large software company would likely result in Sales & Marketing being empowered. Which in turn will likely lead to technical debt (as the company stretches to meet impractical promises), as well as stagnation in Engineering (S&M are delivering on the metrics, so they get promoted). In general, large organizations are like an AI. In this case, making a goal that won't backfire when applied without thought is really difficult.
- nostrademons 8y ago
  That's why the goal in so many large public companies becomes "Make the stock price go up." If the stock price goes up, the executives and everyone else in the company with stock can unload it, take the cash, and let the new shareholders deal with any long-term consequences of their actions. This works great up until it doesn't.

minimaxir 8y ago

When I interviewed for a data analyst position at a known startup, I was asked to define a target KPI for a controlled pseudo-A/B experiment, and I suggested total revenue. The interviewer replied "that's a dumb answer because revenue can be gamed; you should have said # of sales." I was too taken aback to follow up asking why # of sales can't be gamed too.

bsg75 8y ago
Number of sales (or customers) is a vanity metric favored by startups looking to entice investment. Revenue is a metric useful for measuring actual business health. Any chance the interviewer was overly exit eager ?
- minimaxir 8y ago
  It is a startup very well known for being a loss-leader.
rhizome 8y ago
Credit where due, I think this is the first time I've read of anybody being told how they got something wrong in an interview.
- geofft 8y ago
  Why is that credit? Good interview questions don't have a right or wrong answer, they have a right or wrong process of coming up with an answer. Especially for a question like "Pick a metric," almost any relevant metric is defensible and has tradeoffs, and the question should be about how the interviewee analyzes engineering tradeoffs, gathers requirements, defends their initial proposal, and is open to counterarguments. (If there's a correct answer, you can just give them a quiz. If there's a Googleable correct answer, you can just employ them and give them Google.)
abiox 8y ago
> interviewer replied "that's a dumb answer a bit ot, but was that the actual response? personally, stuff like this would have me ending the interview early.

heavenlyblue 8y ago

That’s why organisations need leaders who can change the direction having realised that the current state had degraded to being gamed. Seek revenue, then seek new clients but not revenue, then go on R&D spree. People are not going to be happy, but the organisation is going to be healthy. IMO there must be high-level metrics which define which low-level metrics are valuable today.

usrusr 8y ago
It's getting weirder when you factor in shareholder value: sacrificing the future of the company for looking good in all the fashionable metrics might be exactly the leaders job, depending on how the board is made up.

mwexler 8y ago

But of course, "If you can't measure it, you can't manage it", said by Drucker, says EVERYONE. Hence the cherry picking of preferential KPIs or OKRs. But, actually, Drucker didn't say it. In fact, W. Edwards Deming said it. And what he actually said was "It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.". So, basically, the quote held up as the "reason" behind excessive over-tracking of the wrong thing... is wrong. https://blog.deming.org/2015/08/myth-if-you-cant-measure-it-you-cant-manage-it/ https://blog.deming.org/2015/08/myth-if-you-cant-measure-it-...