4 ms·

Metrics are great for optimizing your business not for deciding which one to build.

TeMPOraL 7y ago
The problem talked about in the article is that it's not easy to express your optimization goals in terms of concrete metrics, and your optimization process will always optimize those metrics, not your real goals.
- ThomPete 7y ago
  Yes, the art is always to pick the right metrics. The problem is when companies choose qualitative and thus interpretational metrics. Just stay to the simple easily trackable and quantitative ones and the make it your job to interpret them qualitatively.
  - TeMPOraL 7y ago
    Most real-life goals can't be easily expressed with few "easily trackable" metrics. Take example from the article, customer satisfaction. That's half of the problem; the other half is that metrics can be gamed, and they'll end up being gamed accidentally or for profit if you're not careful. Like, the easiest way to make a system stable is to make it so painful to use that people don't use it - and if they don't use it, they can't break it. Or, the example of return tracking from the TFA. I read a lot about how data-driven companies measure this and that, often through questionable, privacy-violating measures. What I don't read about is how do these companies ensure the metrics are actually valid - that they're correctly sampling the population[0], or that they're measuring what the authors think they're measuring, or that they're not being misreported or otherwise gamed (very common if the value of a metric impacts someone's career or even workload). -- [0] - e.g. voluntary surveys usually don't, telemetry increasingly doesn't either as more and more people are aware of it and disable it.
    - ThomPete 7y ago
      That's what I am saying. Don't choose things that can't be tracked quantitatively. Track things quantitatively and then make qualitative decisions. At Square, we had "Make Commerce Easy". This was qualitative and thus up for interpretation. It became even vaguer when it became "Economic empowerment". These things can't and shouldn't be tracked as if they are quantitative metrics but instead used as guiding stars. So the trick is just to separate the two and as I said. Use data metrics for optimizing and not for creating or measuring of qualitative goals.
      - TeMPOraL 7y ago
        I see your point now, thanks for the clarification.
      - dragonwriter 7y ago
        > Track things quantitatively and then make qualitative decisions Decisions are often inherently quantitative, e.g., you allocate an actual dollar amount to a budget. Or a project go/no-go is a crisp binary decision. I think you mean don't make decisions directly based on quantitative metrics but on qualitative assessments informed by quantitative metrics.
        ThomPete 7y ago
        Hmm maybe. I don't distinguish that hard the second I get into the qualitative so many factors (like experience) inform the quantitative work. Sometimes the qualitative goals is created because of the quantitative data. What you are saying is certainly also what I mean but I don't think I only mean that but I see what you are saying and yeah it might just be my english.
- dragonwriter 7y ago
  > The problem talked about in the article is that it's not easy to express your optimization goals in terms of concrete metrics, Largely, because what you are wanting to optimize is something like current value of the future income stream of the firm. So a proxy is taken, which inevitably has bias, and you need to understand why hat that big is likely to be a continuously sanity check your results for reason to think that you may being caught in that bias. Which you probably can't measure well (though maybe something has improved since you adopted the proxy), because if you could you'd have a better proxy to start with.
  - TeMPOraL 7y ago
    True, though I was thinking about something else still: unless you're a completely amoral alien mind, even your for-profit goals have some hidden value judgements and preferences attached. Those extra constraints are incredibly hard to capture as explicit goals and thus impossible to properly measure in a quantified way. So if you now turn the optimization knob up to 11 and just let it work for a while, you might be appalled by what it comes up with. Or if not you, the people on the receiving end of your optimizations definitely will.

The problem talked about in the article is that it's not easy to express your optimization goals in terms of concrete metrics, and your optimization process will always optimize those metrics, not your real goals.

ThomPete 7y ago
Yes, the art is always to pick the right metrics. The problem is when companies choose qualitative and thus interpretational metrics. Just stay to the simple easily trackable and quantitative ones and the make it your job to interpret them qualitatively.
- TeMPOraL 7y ago
  Most real-life goals can't be easily expressed with few "easily trackable" metrics. Take example from the article, customer satisfaction. That's half of the problem; the other half is that metrics can be gamed, and they'll end up being gamed accidentally or for profit if you're not careful. Like, the easiest way to make a system stable is to make it so painful to use that people don't use it - and if they don't use it, they can't break it. Or, the example of return tracking from the TFA. I read a lot about how data-driven companies measure this and that, often through questionable, privacy-violating measures. What I don't read about is how do these companies ensure the metrics are actually valid - that they're correctly sampling the population[0], or that they're measuring what the authors think they're measuring, or that they're not being misreported or otherwise gamed (very common if the value of a metric impacts someone's career or even workload). -- [0] - e.g. voluntary surveys usually don't, telemetry increasingly doesn't either as more and more people are aware of it and disable it.
  - ThomPete 7y ago
    That's what I am saying. Don't choose things that can't be tracked quantitatively. Track things quantitatively and then make qualitative decisions. At Square, we had "Make Commerce Easy". This was qualitative and thus up for interpretation. It became even vaguer when it became "Economic empowerment". These things can't and shouldn't be tracked as if they are quantitative metrics but instead used as guiding stars. So the trick is just to separate the two and as I said. Use data metrics for optimizing and not for creating or measuring of qualitative goals.
    - TeMPOraL 7y ago
      I see your point now, thanks for the clarification.
    - dragonwriter 7y ago
      > Track things quantitatively and then make qualitative decisions Decisions are often inherently quantitative, e.g., you allocate an actual dollar amount to a budget. Or a project go/no-go is a crisp binary decision. I think you mean don't make decisions directly based on quantitative metrics but on qualitative assessments informed by quantitative metrics.
      - ThomPete 7y ago
        Hmm maybe. I don't distinguish that hard the second I get into the qualitative so many factors (like experience) inform the quantitative work. Sometimes the qualitative goals is created because of the quantitative data. What you are saying is certainly also what I mean but I don't think I only mean that but I see what you are saying and yeah it might just be my english.
dragonwriter 7y ago
> The problem talked about in the article is that it's not easy to express your optimization goals in terms of concrete metrics, Largely, because what you are wanting to optimize is something like current value of the future income stream of the firm. So a proxy is taken, which inevitably has bias, and you need to understand why hat that big is likely to be a continuously sanity check your results for reason to think that you may being caught in that bias. Which you probably can't measure well (though maybe something has improved since you adopted the proxy), because if you could you'd have a better proxy to start with.
- TeMPOraL 7y ago
  True, though I was thinking about something else still: unless you're a completely amoral alien mind, even your for-profit goals have some hidden value judgements and preferences attached. Those extra constraints are incredibly hard to capture as explicit goals and thus impossible to properly measure in a quantified way. So if you now turn the optimization knob up to 11 and just let it work for a while, you might be appalled by what it comes up with. Or if not you, the people on the receiving end of your optimizations definitely will.