4 ms·

My experience is similar. Even if and when the metrics are calculated properly, there's often some design or business reason put forth as an excuse to ignore th

My experience is similar. Even if and when the metrics are calculated properly, there's often some design or business reason put forth as an excuse to ignore them.

codeptualize 5y ago
It seems that any kind of data is mainly used to confirm the believes people already had and/or justify decisions already made.
- lewisl9029 5y ago
  Not to mention the decision paralysis and change aversion it often introduces into company culture where every change, however trivial or however obviously beneficial, has to first go through a 2-week A/B test which often turns out to be inconclusive anyways, and sometimes takes more eng resources to set up and run than it takes to make the change itself.
  - ssharp 5y ago
    > however trivial Previous testing should give the company at least some baseline understanding of what is trivial and what isn't. The correct way to experiment is certainly not "let's experiment every idea!" > however obviously beneficial If you've been around long enough, you've almost certainly run into dozens of "obviously beneficial" changes that led to poorer performance. Most of what you're describing is issues with poor prioritization, a lack of understanding about your audience, and a culture that has a difficult time making decisions.
sosborn 5y ago
Before any effort goes into something like this I always raise my hand and ask, "If the data shows us something we don't want to see, will we change our strategy? If not, I'd rather put time/effort into other projects." It works about 80% of the time. Metrics are only useful if the organization is actually willing to learn lessons from them.
- kqr 5y ago
  Oh yeah, defining the thresholds and their associated courses of action ahead of time is important to make a good decision. Any time someone wants to measure something, the top two questions should be "what are the lower and upper bounds this value has to exceed for us to do something different?" Very often, it turns out these thresholds for change are so astronomical that nobody thinks we have even the slightest chance of exceeding them. That means the measurement is completely useless. Whatever result we plausibly get, it won't change anything. You can save a lot of time this way!

My experience is similar. Even if and when the metrics are calculated properly, there's often some design or business reason put forth as an excuse to ignore them.

codeptualize 5y ago

It seems that any kind of data is mainly used to confirm the believes people already had and/or justify decisions already made.

lewisl9029 5y ago
Not to mention the decision paralysis and change aversion it often introduces into company culture where every change, however trivial or however obviously beneficial, has to first go through a 2-week A/B test which often turns out to be inconclusive anyways, and sometimes takes more eng resources to set up and run than it takes to make the change itself.
- ssharp 5y ago
  > however trivial Previous testing should give the company at least some baseline understanding of what is trivial and what isn't. The correct way to experiment is certainly not "let's experiment every idea!" > however obviously beneficial If you've been around long enough, you've almost certainly run into dozens of "obviously beneficial" changes that led to poorer performance. Most of what you're describing is issues with poor prioritization, a lack of understanding about your audience, and a culture that has a difficult time making decisions.

sosborn 5y ago

Before any effort goes into something like this I always raise my hand and ask, "If the data shows us something we don't want to see, will we change our strategy? If not, I'd rather put time/effort into other projects." It works about 80% of the time. Metrics are only useful if the organization is actually willing to learn lessons from them.

kqr 5y ago
Oh yeah, defining the thresholds and their associated courses of action ahead of time is important to make a good decision. Any time someone wants to measure something, the top two questions should be "what are the lower and upper bounds this value has to exceed for us to do something different?" Very often, it turns out these thresholds for change are so astronomical that nobody thinks we have even the slightest chance of exceeding them. That means the measurement is completely useless. Whatever result we plausibly get, it won't change anything. You can save a lot of time this way!