4 ms·

> are irrationally attached to P values This is actually a terrible problem. Most people, I think, do this because of nervous cluelessness: they are sure there

> are irrationally attached to P values This is actually a terrible problem. Most people, I think, do this because of nervous cluelessness: they are sure there is some theory that says what it means, but they aren't sure they understand it, so they insist on following some strict rules, whether they make sense or not. I think the idea is that if you follow a strict rule given in a textbook (even if wrong or inapplicable), that makes people feel safer than if they try to figure the rule out for themselves. At least if they follow a rule, they aren't personally responsible for the outcome. It can sometimes be quite difficult to explain why a certain rule should not be applied, because the same reasons for why they didn't understand the rule in the first place also make it difficult to understand explanations given by someone else.

wodenokoto 12y ago
> so they insist on following some strict rules, whether they make sense or not. I took my bachelor in business and we were always told to make rational business decision. That is, informed decisions, based on data. As for actual statistics, we were told 2 rules of thumbs: You need at least 30 people in a group before you can make meaningful statistics and it is significant if P < 0.005
- RA_Fisher 12y ago
  Unfortunately that advice is hugely misleading. The reason is that the p-value essentially tells you how repeatable your analysis will be, but doesn't say anything about how beneficial making the decision might be. Second, there is a relationship between how large the difference between outcomes are and how many people you need. So if taking your conversion rate from 10 to 15% would be really helpful to your business, you might need say thousands of people to notice that difference statistically.
  - cgearhart 12y ago
    > The reason is that the p-value essentially tells you how repeatable your analysis will be Your statement contradicts the introductory paragraph of the article: > "...the outcomes of these tests are mistakenly believed to yield the following information:... the probability that an initial finding will replicate; ..." The p-value is the likelihood that the observed effect is due only to chance, not a measure of the repeatability of a result.
    - RA_Fisher 12y ago
      Ah yes, you're right! I made a statement about likelihood!
- wodenokoto 12y ago
  This was meant as an example of how crazily statistics is taught and why people end up just following strict rules.

wodenokoto 12y ago

> so they insist on following some strict rules, whether they make sense or not. I took my bachelor in business and we were always told to make rational business decision. That is, informed decisions, based on data. As for actual statistics, we were told 2 rules of thumbs: You need at least 30 people in a group before you can make meaningful statistics and it is significant if P < 0.005

RA_Fisher 12y ago
Unfortunately that advice is hugely misleading. The reason is that the p-value essentially tells you how repeatable your analysis will be, but doesn't say anything about how beneficial making the decision might be. Second, there is a relationship between how large the difference between outcomes are and how many people you need. So if taking your conversion rate from 10 to 15% would be really helpful to your business, you might need say thousands of people to notice that difference statistically.
- cgearhart 12y ago
  > The reason is that the p-value essentially tells you how repeatable your analysis will be Your statement contradicts the introductory paragraph of the article: > "...the outcomes of these tests are mistakenly believed to yield the following information:... the probability that an initial finding will replicate; ..." The p-value is the likelihood that the observed effect is due only to chance, not a measure of the repeatability of a result.
  - RA_Fisher 12y ago
    Ah yes, you're right! I made a statement about likelihood!
wodenokoto 12y ago
This was meant as an example of how crazily statistics is taught and why people end up just following strict rules.