4 ms·

This post is amazing. When I see a post "P < 0.05 Considered Harmful" I think OK now they are going to talk about maybe Bonferroni vs. other multiple hypothesi

by ftxbro 4y ago

This post is amazing. When I see a post "P < 0.05 Considered Harmful" I think OK now they are going to talk about maybe Bonferroni vs. other multiple hypothesis corrections if they are a frequentist or otherwise they are going to try to explain Bayesian things. But no, this one isn't from a frequentist, or from a Bayesian. It's from a techbro whose solution isn't any kind of multiple hypothesis correction or getting Bayes-pilled, it's to say "Why not just admit you want something akin to p = 0.25 in the first place?" for 'ship criteria' in the only stats context he appears to know which is A/B testing, talking about Maslow hierarchy and namedropping Hula and Netflix. It's seriously like some Silicon Valley parody. Wait is it actually a satire blog?

whimsicalism 4y ago
> "Why not just admit you want something akin to p = 0.25 in the first place?" for 'ship criteria' That's culture shock for me - I guess this is why I don't work at startups.
- bookish 4y ago
  It all depends on what you're experimenting on. I do think there are many teams out there who are (in effect) making decisions with less certainty than this, but they wouldn't want to actually quantify it.
- mlyle 4y ago
  p=0.2 doesn't work too well for ship criteria for medicine. p=0.2 for "this reordering of the landing text improves the rate conversion events" is fine. People make changes based on less information all the time. Waiting for certainty has its own expenses.
  - mike_hearn 4y ago
    Not that fine. The author claims that neutral changes are not costly for users so the only reason to avoid them is to avoid wasting time/money. But that's not really true. Pointless churn annoys users. If your P threshold isn't low enough then you can get stuck in an endless treadmill of making changes that you thought would have benefit but which don't actually do anything because your threshold for something being considered significant is too loose.
    - mlyle 4y ago
      > Pointless churn annoys users. Sure, excessive rate of change is bad. Even if it's all validated and well-proven. > If your P threshold isn't low enough then you can get stuck in an endless treadmill of making changes To measure P, I have to make the change. There's plenty of situations where accumulating enough data to meet a p<0.05 threshold would take months. If my prior is that it's a reasonably good change and has a decent chance to be helpful, and then I take a measurement that makes that roughly 5x more likely... it's OK to push the button. There are many decisions in business that have to be made with much less information or statistical proof than this.
thebestgamers 4y ago
[flagged]
SantalBlush 4y ago
It made the front page because of the title. Bashing p-values is a formula for upvotes, regardless of the content. A lot of people associate p-values with p-hacking and replication crisis, which they read about on HN a few times while sitting on the can. Therefore, in their expert opinions, p-values are bad.

by ftxbro 4y ago

whimsicalism 4y ago

> "Why not just admit you want something akin to p = 0.25 in the first place?" for 'ship criteria' That's culture shock for me - I guess this is why I don't work at startups.

bookish 4y ago
It all depends on what you're experimenting on. I do think there are many teams out there who are (in effect) making decisions with less certainty than this, but they wouldn't want to actually quantify it.
mlyle 4y ago
p=0.2 doesn't work too well for ship criteria for medicine. p=0.2 for "this reordering of the landing text improves the rate conversion events" is fine. People make changes based on less information all the time. Waiting for certainty has its own expenses.
- mike_hearn 4y ago
  Not that fine. The author claims that neutral changes are not costly for users so the only reason to avoid them is to avoid wasting time/money. But that's not really true. Pointless churn annoys users. If your P threshold isn't low enough then you can get stuck in an endless treadmill of making changes that you thought would have benefit but which don't actually do anything because your threshold for something being considered significant is too loose.
  - mlyle 4y ago
    > Pointless churn annoys users. Sure, excessive rate of change is bad. Even if it's all validated and well-proven. > If your P threshold isn't low enough then you can get stuck in an endless treadmill of making changes To measure P, I have to make the change. There's plenty of situations where accumulating enough data to meet a p<0.05 threshold would take months. If my prior is that it's a reasonably good change and has a decent chance to be helpful, and then I take a measurement that makes that roughly 5x more likely... it's OK to push the button. There are many decisions in business that have to be made with much less information or statistical proof than this.

thebestgamers 4y ago

[flagged]

SantalBlush 4y ago

It made the front page because of the title. Bashing p-values is a formula for upvotes, regardless of the content. A lot of people associate p-values with p-hacking and replication crisis, which they read about on HN a few times while sitting on the can. Therefore, in their expert opinions, p-values are bad.