4 ms·

> "3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold. There is no lower thre

by wdewind 11y ago

> "3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold. There is no lower threshold at which the data becomes non-predictive?

carbocation 11y ago
More likely they mean the opposite: that a statistically significant P value (by whatever threshold you decide to use) should not be used, by itself, to drive policy decisions. Internally, the effect size still matters. Externally, there are numerous other factors that should drive decisionmaking.
- wdewind 11y ago
  Right, but I guess what I'm getting at is frequently we see people doing even worse: making policy decisions based on data that doesn't even hit a minimum threshold for acceptability. I totally agree that people shouldn't make policy decisions solely because the data supports it, but frequently you see people make decisions based on data indicating something without actually being statistically significant. That feels like a bigger problem to me than people actually getting good data but then using it too confidently, which I have rarely if ever seen. TLDR: People tend to make policy decisions based on data even when the data is basically useless, I'm more worried about that then blindly following good data.
  - snowwrestler 11y ago
    > That feels like a bigger problem to me than people actually getting good data but then using it too confidently P value alone does not tell you whether you have good data. It just tells you how well a particular model fits the data that you have. It won't (and can't) tell you if your data set is missing data that would alter the P value were it to be included. P value alone is not enough to say "good" or "bad." That's why this statement matters.
    - wdewind 11y ago
      I'm not sure I agree. If P-value is really low then it implies either the existing data set is bad or your new data set is bad. Either way you haven't used data to get to a confident decision. Put another way: tons of stuff in prod dev gets done without p-values entirely because it's the first time you have any data at all about something. I'm questioning whether this is hugely valuable and obviously better than using my intuition or not, since we really don't have a good understanding of the baseline.
      - deleted 11y ago
        [deleted]
      - deleted 11y ago
        [deleted]
  - skybrian 11y ago
    One issue is that if you have a large effect that's consistently and easily reproduced, you don't actually need very accurate measurements or a statistical analysis at all. So any minimum standard would need to take into account. Another issue is that science is expensive and we need to make decisions all the time whether there is any science backing them or not. So what do you do if there's no science that meets the minimum standards?
    - wdewind 11y ago
      > One issue is that if you have a large effect that's consistently and easily reproduced, you don't actually need very accurate measurements or a statistical analysis at all. I mostly agree with this, but you never have that in a difficult decision. > Another issue is that science is expensive and we need to make decisions all the time whether there is any science backing them or not. So what do you do if there's no science that meets the minimum standards? Right! My entire point is that maybe data driven decision making isn't as useful as we think because it mostly doesn't hit scientific levels of accuracy. Maybe there are other, more intuitive systems, that make as much if not more sense, in absence of good data. How would we even know either way? We can't! Regardless I'd love for the discussion to be "this is the best we can do" rather than "yes we did A/B testing so we know it's true!!!"
      - thaumasiotes 11y ago
        >> One issue is that if you have a large effect that's consistently and easily reproduced, you don't actually need very accurate measurements or a statistical analysis at all. > I mostly agree with this, but you never have that in a difficult decision. Consider the problem "should we cure congenital deafness in infants?" To Deaf Community Leaders, this isn't a difficult decision at all; they are strongly against curing deaf children because it makes their power base smaller. To deaf parents of deaf children, this is a tricky choice. Curing the child's deafness dramatically improves its prospects in life, but it also inevitably cuts the child off from the parents' community. You have to choose between how good you want your child's future to be, and how close you'd like your relationship with them to be. To hearing parents of deaf children, this is again a trivial choice; obviously you'd cure the child. BUT, curing deafness is definitely (1) a large effect that is (2) easily reproduced.
  - shimn 11y ago
    > should not be based only on whether a p-value passes a specific threshold. It seems as though you'd missed the "only" in the clause. Further, they say nothing w.r.t. "does not pass" (aka: not hitting minimum threshold). Both your complaint and their point are orthogonal.
deleted 11y ago
[deleted]

by wdewind 11y ago

carbocation 11y ago

More likely they mean the opposite: that a statistically significant P value (by whatever threshold you decide to use) should not be used, by itself, to drive policy decisions. Internally, the effect size still matters. Externally, there are numerous other factors that should drive decisionmaking.

wdewind 11y ago
Right, but I guess what I'm getting at is frequently we see people doing even worse: making policy decisions based on data that doesn't even hit a minimum threshold for acceptability. I totally agree that people shouldn't make policy decisions solely because the data supports it, but frequently you see people make decisions based on data indicating something without actually being statistically significant. That feels like a bigger problem to me than people actually getting good data but then using it too confidently, which I have rarely if ever seen. TLDR: People tend to make policy decisions based on data even when the data is basically useless, I'm more worried about that then blindly following good data.
- snowwrestler 11y ago
  > That feels like a bigger problem to me than people actually getting good data but then using it too confidently P value alone does not tell you whether you have good data. It just tells you how well a particular model fits the data that you have. It won't (and can't) tell you if your data set is missing data that would alter the P value were it to be included. P value alone is not enough to say "good" or "bad." That's why this statement matters.
  - wdewind 11y ago
    I'm not sure I agree. If P-value is really low then it implies either the existing data set is bad or your new data set is bad. Either way you haven't used data to get to a confident decision. Put another way: tons of stuff in prod dev gets done without p-values entirely because it's the first time you have any data at all about something. I'm questioning whether this is hugely valuable and obviously better than using my intuition or not, since we really don't have a good understanding of the baseline.
    - deleted 11y ago
      [deleted]
    - deleted 11y ago
      [deleted]
- skybrian 11y ago
  One issue is that if you have a large effect that's consistently and easily reproduced, you don't actually need very accurate measurements or a statistical analysis at all. So any minimum standard would need to take into account. Another issue is that science is expensive and we need to make decisions all the time whether there is any science backing them or not. So what do you do if there's no science that meets the minimum standards?
  - wdewind 11y ago
    > One issue is that if you have a large effect that's consistently and easily reproduced, you don't actually need very accurate measurements or a statistical analysis at all. I mostly agree with this, but you never have that in a difficult decision. > Another issue is that science is expensive and we need to make decisions all the time whether there is any science backing them or not. So what do you do if there's no science that meets the minimum standards? Right! My entire point is that maybe data driven decision making isn't as useful as we think because it mostly doesn't hit scientific levels of accuracy. Maybe there are other, more intuitive systems, that make as much if not more sense, in absence of good data. How would we even know either way? We can't! Regardless I'd love for the discussion to be "this is the best we can do" rather than "yes we did A/B testing so we know it's true!!!"
    - thaumasiotes 11y ago
      >> One issue is that if you have a large effect that's consistently and easily reproduced, you don't actually need very accurate measurements or a statistical analysis at all. > I mostly agree with this, but you never have that in a difficult decision. Consider the problem "should we cure congenital deafness in infants?" To Deaf Community Leaders, this isn't a difficult decision at all; they are strongly against curing deaf children because it makes their power base smaller. To deaf parents of deaf children, this is a tricky choice. Curing the child's deafness dramatically improves its prospects in life, but it also inevitably cuts the child off from the parents' community. You have to choose between how good you want your child's future to be, and how close you'd like your relationship with them to be. To hearing parents of deaf children, this is again a trivial choice; obviously you'd cure the child. BUT, curing deafness is definitely (1) a large effect that is (2) easily reproduced.
- shimn 11y ago
  > should not be based only on whether a p-value passes a specific threshold. It seems as though you'd missed the "only" in the clause. Further, they say nothing w.r.t. "does not pass" (aka: not hitting minimum threshold). Both your complaint and their point are orthogonal.

deleted 11y ago

[deleted]