3 ms·

This is mostly right, but you are still thinking of these rejections as "false positives" for some reason. They are real deviations from the null hypothesis ("t

by nonbel 7y ago

This is mostly right, but you are still thinking of these rejections as "false positives" for some reason. They are real deviations from the null hypothesis ("true positives"). The problem is the user didn't test the null model they wanted, it is 100% user error.

dwaltrip 7y ago
Can you explain that last sentence? What is a valid null model if everything is correlated?
- nonbel 7y ago
  A model of whatever process you think generated the data. EDIT: I guess I should say that the concept of testing a "null model" without interpreting the fit relative to other models is wrong to begin with. You need to use Bayes' rule and determine: p(H[0]|D) = p(H[0])p(D|H[0])/sum(p(H[0:n])*p(D|H[0:n])) Lots of stuff wrong with what has been standard stats for the last 70 years, it literally amounts to stringing together a bunch of fallacies and makes no sense at all.
  - dwaltrip 7y ago
    Thanks for the response. Do you know of any you good blog posts or articles that dive into this a bit more? It looks very interesting.
    - nonbel 7y ago
      This is the best description of the main problem (testing your own vs some default hypothesis) I have seen: Paul E. Meehl, "Theory-Testing in Psychology and Physics: A Methodological Paradox," Philosophy of Science 34, no. 2 (Jun., 1967): 103-115. https://doi.org/10.1086/288135 https://doi.org/10.1086/288135 Free download here: www.fisme.science.uu.nl/staff/christianb/downloads/meehl1967.pdf Andrew Gelman (andrewgelman.com) has a great blog that often touches on this issue.
      - dwaltrip 7y ago
        Thanks a bunch for sharing, I appreciate it. I'll add these resources to my reading list. I may also pass it along to my brother who getting a graduate degree in psych :)
    - nonbel 7y ago
      https://statmodeling.stat.columbia.edu/2019/03/20/retire-statistical-significance-the-discussion/ https://statmodeling.stat.columbia.edu/2019/03/20/retire-sta...

by nonbel 7y ago

dwaltrip 7y ago

Can you explain that last sentence? What is a valid null model if everything is correlated?

nonbel 7y ago
A model of whatever process you think generated the data. EDIT: I guess I should say that the concept of testing a "null model" without interpreting the fit relative to other models is wrong to begin with. You need to use Bayes' rule and determine: p(H[0]|D) = p(H[0])p(D|H[0])/sum(p(H[0:n])*p(D|H[0:n])) Lots of stuff wrong with what has been standard stats for the last 70 years, it literally amounts to stringing together a bunch of fallacies and makes no sense at all.
- dwaltrip 7y ago
  Thanks for the response. Do you know of any you good blog posts or articles that dive into this a bit more? It looks very interesting.
  - nonbel 7y ago
    This is the best description of the main problem (testing your own vs some default hypothesis) I have seen: Paul E. Meehl, "Theory-Testing in Psychology and Physics: A Methodological Paradox," Philosophy of Science 34, no. 2 (Jun., 1967): 103-115. https://doi.org/10.1086/288135 https://doi.org/10.1086/288135 Free download here: www.fisme.science.uu.nl/staff/christianb/downloads/meehl1967.pdf Andrew Gelman (andrewgelman.com) has a great blog that often touches on this issue.
    - dwaltrip 7y ago
      Thanks a bunch for sharing, I appreciate it. I'll add these resources to my reading list. I may also pass it along to my brother who getting a graduate degree in psych :)
  - nonbel 7y ago
    https://statmodeling.stat.columbia.edu/2019/03/20/retire-statistical-significance-the-discussion/ https://statmodeling.stat.columbia.edu/2019/03/20/retire-sta...